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1 1th  Annual  Systems  Engineering  Conference 

San  Diego,  CA 

20-23  October  2008 


Agenda 


TUESDAY.  21  OCTOBER  2008 


Keynote  Addresses: 

•  HON  Charles  McQueary,  Director,  Operational  Test  &  Evaluation; 

Plenary  Session:  Executive  Panel 

Moderator: 

Ms.  Kristin  Baldwin,  Deputy  Director,  Software  Engineering  &  System  Assurance 
Panelists: 

•  Mr.  Terry  Jaggers,  Director,  SAF/AQR  (Science,  Technology  &  Engineering) 

•  Mr.  Carl  Siel,  Chief  Systems  Engineer;  ASN(RDA)CHENG 

•  Mr.  Ross  Guckert,  Assistant  Deputy,  Acquisition  &  Systems  Integration  ASA(ALT) 
Luncheon  with  Speaker  in  the  Regatta  Pavilion 

•  Dr.  Ronald  Jost,  Deputy  Assistant  Secretary  of  Defense,  C3,  Space  &  Spectrum 

BAYVIEW  III:  SYSTEMS  ENGINEERING  EFFECTIVENESS 
Session  2C1 


o  7099-  DoD’s  Systems  and  Engineering  Revitalization  Efforts-  An  Update  Mr.  Nicholas  M.  Torelli,  OSD/SSE/ED 
o  7475  -  The  Effectiveness  of  Systems  Engineering  on  Federal  (DoD)  System  Development  Programs  -  Update  2008,  Mr.  Ken  Ptack 
o  7153-  Systems  Engineering  Plan  (SEP)  and  Systems  Engineering  Management  Plan  (SEMP)  Unification  Mr.  Chet  Bracuto,  OSD 
°  Naval  Power  21  Integration  &  Interoperability  Improvement,  Mr.  Kevin  Smith 
o  7089  -  Systems  Engineering  for  Systems  of  Systems,  Dr.  Judith  Dahmann,  The  MITRE  Corporation 

BAYVIEW  II:  TEST  &  EVALUATION  IN  SYSTEMS  ENGINEERING 
Session  2C2 

o  7100-  Implementation  of  the  2007  Developmental  Test  &  Evaluation  Defense  Science  Board  Results:  Mr.  Chris  DiPetto,  OUSD/SSR/ 
o  7101  -  Test  and  Evaluation  Value  Metrics  at  Acquisition  Decision  Points:  Ms.  Darlene  Mosser-Kerner,  OUSD/SSE/DTE 
°  6979  -  Integration  of  Software  Intensive  Systems:  Mr.  Tom  Wissink,  Lockheed  Martin 
o  6996  -  Modeling  &  Simulation  in  the  Test  &  Evaluation  Master  Plan,  Mr.  Michael  Truelove 
o  7 1 03  -  “New. . .  .Improved”  Test  &  Evaluation  Master  Plan,  Ms.  Darlene  Mosser-Kerner 
o  7290  -  Mission  Based  T&E  Strategy,  Mr.  Chris  Wilcox 

BAYVIEW  I:  PROGRAM  MANAGEMENT 
Session  2C3 

°  7096  -  New  Acquisition  Policy  and  Its  Impact  on  Defense  Systems  Engineering:  Ms.  Sharon  Vannucci,  ODUSD/SSE/ED 
o  6919-  Improving  the  Quality  of  DoD  Weapon  Systems:  Ms.  Cheryl  K.  Andrew,  U.S.  Government  Accountability  Office 
o  An  Air  Force  S&T  Directorate’s  View  on  Applying  Systems  Engineering  Principles  to  its  Programs 

o  High  Confidence  Technology  Transition  Planning  Through  the  Use  of  Stage-Gates  (TD-13),  Dr.  Claudia  Kropas-Hughes,  HQ,  AFMC 
°  7002  -  Systems  Engineering  Re-vitalization  at  the  Defense  Contract  Management  Agency:  Mr.  Lawrence  F.  Cianciolo,  Defense  Contract  Management 
Agency 

MISSION  I  SYSTEM  SAFETY-  ESOH  &  HSI 
Session  2C4 
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o  6997  -  Human  Systems  Integration  and  Model  Based  Systems  Engineering:  Dr.  Abraham  W.  Meilich,  Lockheed  Martin 

o  7084  -  Human  Reliability  Analysis  and  the  Advanced  Man  Portable  Air  Defense  System:  A  Case  Study:  Mr.  Christopher  A.  Brown,  Naval  Surface 
Warfare  Center,  Crane 

o  7092-  Systems  Engineering  to  Ensure  Aircraft  Airworthiness,  Mr.  Jim  Miller 
o  7161  -  ESOH  In  Acquisition  OSD  Expectations  For  Implementing  DODI  5000.02,  Ms.  Karen  Gill 
©  ESOH  Challenges  in  Commissioning  an  Aircraft  Carrier,  Mr.  Doug  Parrish,  Booz  Allen  Hamilton 

MISSION  II  MODELING  &  SIMULATION 
Session  2C5 

o  7172  -  Execution  of  the  Acquisition  M&S  Master  Plan-  A  Progress  Report:  Mr.  James  W..  Hollenbach,  Simulation  Strategies,  Inc. 
o  Update  on  Survey  on  Modeling  and  Simulation  Support  for  the  Systems  Engineering  of  Systems  of  Systems,  Ms.  Judith  Dahmann,  Simulation 
Strategies,  Inc 

o  7440  -  Synchronizing  Modeling  and  Simulation  Plans  Across  Navy  Acquisition:  Dr.  Ivar  Oswalt,  VisiTech 
o  7085  -  Modeling  and  Simulation  Resource  Reuse  Business  Model:  Mr.  Dennis  P.  Shea,  Center  For  Naval  Analyses 

o  Joint  Rapid  Scenario  Generation  (JRSG)  System  Engineering,  Mr.  Ralph  O’Connell,  US  Joint  Forces  Command,  Joint  Capability  Development  (J8) 
o  Cross-Command  Collaboration  Effort  (3CE) 

MISSION  III:  NET  CENTRIC  OPERATIONS 
Session  2C6 

o  7461 -Network  Centric  Engineering  use  of  the  NCOIC  (Network  Centric  Operations  Industry  Consortium)  Processes  and  Tools  in  a  Logistics  Example: 
Mr.  Thomas  M.  Dlugolecki,  SenseResponder  LLC 

o  7128  -  Changing  the  Value  Equation  in  Engineering  and  Acquisition  to  Align  Systems  of  Systems  with  Dynamic  Mission  Needs:  Mr.  Philip  J.  Boxer, 
Software  Engineering  Institute 

°  7341  -  Crucial  Factors  in  the  Design  of  Net-Centric  Systems:  Dr.  David  Hernandez,  Tactronics  Holdings,  LLC 
o  7330  -  Creating  a  Systems  Architecture  for  an  SOA-based  IT  System  as  Part  of  a  Systems  Engineering  Process,  Mr.  Robert  S.  Elinger 
o  A  Service-Oriented  Architecture  (SOA)  Business  Model  for  the  U.S.  Department  of  Defense  (DoD) 

PALM  I:  REQUIREMENTS  DEVELOPMENT  &  MANAGEMENT 
Session  2C7 

o  7444-  Acquisitions  Requirements  of  Capabilities  in  a  Netcentric  Enterprise  -  Creating  a  Capabilities  Engineering  Framework:  Mr.Jack  M.  Van  Kirk, 
SFAE-AV-AS 

o  7138-  Implications  of  Capability-based  Planning  on  Requirements  Engineering:  Mr.  Leonard  Sadauskas,  DoD  CIO,  IT  Investment  &  Commercial 
Policy 

o  7191-  System  Concept  of  Operations:  Standards,  Practices,  and  Reality:  Ms.  Nicole  Roberts,  L-3  Communications 
o  7066  -  Two-Step  Methodology  to  Reduce  Software  System  Requirements  Defects,  Mr.  Robert  J.  Kosman 
°  7451  -  Why  Design  for  Testability  Sooner?,  Mr.  Bruce  Bardell,  BAE  Systems 

o  7399  -  The  Challenges  of  Requirements  Decomposition,  Ms.  Eliza  Siu,  Northrop  Grumman  Corporation 

PALM  II:  SOFTWARE 
Session  2C8 
Panel 

o  7137  -  DoD  Software  Engineering  and  System  Assurance:  Moderator:  Ms.  Kristen  J.  Baldwin,  Systems  and  Software  Engineering 
°  7139  -  A  Framework  for  Integrating  Systems  and  Software  Engineering:  Dr.  Richard  Turner,  Stevens  Institute  of  Technology 
°  7041  -  Software  Process  Improvement  for  Acquisition  of  Naval  Software  Intensive  Systems:  Mr.  Carl  Siel,  U.S.  Navy,  ASN  (RDA)  CHENG 
o  71 19  -  Architecting  Systems  to  Meet  Expectations  -  Managing  Quality  Characteristics  To  Reduce  Risk,  Mr.  Paul  R.  Croll,  CSC 
o  7156  -  New  Concepts  and  Trends  -  How  Future  Trends  in  Systems  and  Software  Technology  Bode  Well  for  Enabling  Improved  Acquisition  and 
Performance  in  Defense  Systems,  Dr.  Kenneth  E.  Nidiffer 

°  7239  -  Systems  and  Software  Design  Principles  for  Large-Scale  Mission-Critical  Embedded  Products  from  Aerospace  and  Financial  Problems 
Domains,  Mr.  Rick  Selby,  Northrop  Grumman  Space  Technology 

WEDNESDAY.  22  OCTOBER  2008 


Luncheon  with  Speaker  in  the  Regatta  Pavilion 

°  Ms.  Shannon  Cunniff,  Director,  Emerging  Containments:  Office  of  Under  Secretary  of  Defense  (Installations  and  Environment) 

BAYVIEW  III:  SYSTEMS  ENGINEERING  EFFECTIVENESS 
Session  3A1 

o  7405  -  Systems  Engineering:  Application  in  Complex  Organizations:  Mr.  Kevin  Roney,  Booz  Allen  Hamilton 
o  7065  -  Establishing  a  Systems  Engineering  Center  of  Excellence  in  PEO  Ground  Combat  Systems:  Mr.  Michael  H.  Phillips,  Jacobs 
o  7423-  Systems  Engineering  Capability  Development:  Mr.  Edward  Andres,  TARDEC 

Session  3B1 

°  7436-  A  Process  Decision  Table  for  Integrated  Systems  and  Software  Engineering:  Dr.  Barry  Boehm,  USC-CSSE 
o  7190  -  A  Tool  to  Enhance  Systems  Engineering  Planning:  Ms.  Sue  O’Brien,  The  University  of  Alabama  in  Huntsville 
o  6945-  The  Role  of  Chaos  and  Complexity  in  Systems  Development:  Dr.  Robert  J.  Monson,  Lockheed  Martin 

Session  3C1 
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°  6878  -  Reduction  of  Total  Ownership  Costs  (R-TOC)  and  Value  Engineering  (VE)  in  Defense  System’s  Life  Cycle:  Mr.  Chet  Bracuto,  OSD 
o  7007  -  Using  Performance-Based  Earned  Value(R)  for  Measuring  Systems  Engineering  Effectiveness:  Dr.  Ronald  S.  Carson,  Boeing 
o  7017-KBAD-  A  Cost-Effective  Way  to  Conduct  Design  and  Analysis:  Dr.  Steven  Dam,  Systems  and  Proposal  Engineering  Company 
o  6886  -  Air  Force  Systems  Engineering  Assessment  Model,  Mr.  Randy  Bullard 
o  7030  -  Defining  100  Best  Practices  for  SE,  Mr.  Ian  Talbot,  AAC/EN 

o  7204  -  Advancing  Systems  Engineering  Practice  within  the  Department  of  Defense:  Overview  of  DoD’s  Newest  University  Affiliated  Research  Center 
(UARC),  Ms.  Sharon  Vannucci,  ODUSD 
o  7093  -  Systems  Engineering  Performance  Measures,  Mr.  Jim  Miller 

BAYVIEW  II:  TEST  &  EVALUATION  IN  SYSTEMS  ENGINEERING 
Session  3A2 

o  6937  -  Systems  Engineering  for  Testing  in  a  Joint  Mission  Environment:  Mr.  Earl  Reyes,  OSD/JTEM 
°  7209-  Joint  Mission  Environment  Test  Capability  (JMETC):  Mr.  Chip  Ferguson,  JMETC 
o  735 1  -  End  to  End  System  Test  Architecture:  Dr.  Masuma  Ahmed,  Lockheed  Martin 

Session  3B2 

o  701 1  -  Implementing  a  Methodology  to  Incorporate  Operational  Realism  in  CONOPS  &  Testing:  Mr.  William  R.  Lyders,  ASSETT,  Inc. 
o  6928  -  The  Role  of  T&E  in  the  Requirements  Process  for  System  of  Systems:  Mr.  Walter  C.  Reel,  Naval  Surface  Warfare  Center  -  Dahlgren 
o  7372  -  Integrated  T&E  Process  and  Tools  in  the  Joint  High  Speed  Vessel  Program:  Mr.  Stephen  F.  Randolph,  Alion  Science  and  Technology 

BAYVIEW  II:  BEST  PRACTICES  &  STANDARDIZATION 
Session  3C2 

o  6874  -  Why  CMMI  Isn’t  Enough:  Ms.  Anita  Carleton,  Software  Engineering  Institute 
o  6888  -  Value  Engineering:  Enhance  DMSMS  Solutions:  Dr.  Jay  Mandelbaum,  Institute  for  Defense  Analysis 

o  7761-  Applying  Business  Process  Modeling  to  Develop  Systems  Engineering  Guidance  for  New  DoD  Acquisition  Regulations:  Dr.  Judith  Dahmann, 
OSD 

Session  3D2 

°  7003  -  How  to  Specify  Applicable  Documents:  Mr.  James  R.  van  Gaasbeek,  Northrop  Grumman 

o  7014  -  Systems  Engineering  in  the  Science  and  Technology  Environment  -  Best  Practices  and  other  Lessons  Learned  from  the  Air  Force  Research 
Laboratory:  Mr.  William  P.  Doyle,  General  Dynamics 

o  7031 -Lessons  Learned  Doing  Systems  Engineering  Assessments  on  the  Government:  Mr.  Ian  Talbot,  AAC/EN 

BAYVIEW  I:  PROGRAM  MANAGEMENT 
Session  3A3 

°  7438  -  The  Incremental  Commitment  Model  and  Competitive  Prototyping:  Dr.  Barry  Boehm,  USC 

o  7070  -  An  Integrated,  Knowledge-based  Approach  to  Developing  Weapon  System  Business  Cases  could  Improve  Acquisition  Outcomes:  Mr.  Travis  J. 
Masters,  U.S.  Government  Accountability  Office 

o  7258  -  Joint  Service  Safety  Testing  Study  Phase  II  Final  Presentation,  Ms.  Paige  V.  Ripani,  Booz  Allen  Hamilton 

Session  3B3 

o  7340  -  “Integrated  Management  Operating  Model  (iMOM)”,  An  E-2D  Advanced  Hawkeye  SD&D  Program  Case  Study:  Mr.  Douglas  J.  Shaffer, 
Northrop  Grumman 

o  7269-  Closing  the  Gap  Between  Systems  Engineering  and  Project  Management:  Mr.  Robert  W.  Ferguson,  Software  Engineering  Institute 
o  7349-  The  Death  of  Rish  Management:  Mr.  Michael  P.  Gaydar,  Naval  Air  Systems  Command 

Session  3C3 

o  7095  -  Evaluating  Complex  System  Development  Maturity-  The  Creation  and  Implementation  of  a  System  Readiness  Level  for  Defense  Acquisition 
Programs:  Mr.  Eric  Forbes,  Northrop  Grumman 

o  7023-  Program  Management  of  Concurrently  Developed  Complex  Systems  -  Lessons  Learned:  Mr.  Alexander  Polack,  The  Aerospace  Corporation 

Session  3D3 

°  7385  -  Enabling  More  Effective  Weapons  Systems  Acquisition  and  Sustainment  through  an  Enterprise  Approach:  Mr.  John  Stewart,  Oracle 
o  7462  -  Applying  the  Tenets  of  Military  Planning  and  Execution  to  Project  and  Systems  Engineering  Management:  Mr.  Philip  Lindeman,  SAIC 
o  7479  -  360  Degree  View  of  the  Technology,  Strategy  and  Business:  Mr.  Min-Gu  Lee,  Lockheed  Martin 


MISSION  I:  SYSTEM  SAFETY-  ESOH  &  HSI 
Session  3B4 

o  721 1  -  Defining  a  Generic  Hazard  Tracking  Database  for  Future  Programs:  Mr.  Jeff  Walker,  Booz  Allen  Hamilton 
o  7215  -  DoD  Energy  Demand:  Addressing  the  Unintended  Consequences:  Mr.  Thomas  Morehouse,  Booz  Allen  Hamilton 
°  7258  -  Joint  Service  Safety  Testing  Study:  Ms.  Paige  Ripani,  Booz  Allen  Hamilton 

Session  3C4 
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o  Update  on  Revisions  to  MIL-STD  882:  Mr.  Robert  “Bob”  Smith,  Booz  Allen  Hamilton 


MISSION  II:  MODELING  &  SIMULATION 
Session  3A5 

o  7347  -  Deployment  of  SysML  in  Tools  and  Architectures:  an  Industry  Perspective:  Mr.  Rick  Steiner,  Raytheon 

°  7073  -  Standardized  Documentation  for  Verification,  Validation,  and  Accreditation  —  An  Update  to  the  Systems  Engineering  Community:  Mr.  Kevin 
Charlow,  Space  and  Warfare  Systems  Center-Charleston 

o  7052  -  Architecture  and  Model  Based  Systems  Engineering  for  Lean  Results:  Mr.  Tim  Olson,  Lean  Solutions  Institute,  Inc. 

Session  3B5 

o  7026  -  Rapid  Assessment  Approach  Using  Commander’s  Intent  to  Identify  Promising  Force  Structure  Architectures  for  System  Trade  Studies:  Mr. 
David  A.  Blancett,  Northrup  Grumman 

°  7082  -  Domain  Modeling:  A  Roadmap  to  Convergence:  Mr.  Nathaniel  C.  Homer,  The  Johns  Hopkins  University  Applied  Physics  Laboratory 
©  7364  -  Predictive  Modeling:  Principles  and  Practice:  Dr.  Rick  Hefner,  Northrop  Gmmman 

Session  3C5 

o  7144  -  Systems  Engineering  Analysis  of  Threat  Reduction  Systems  using  a  Collaborative  Constmctive  Simulation  Environment:  Dr.  James  E. 

Coolahan,  Johns  Hopkins  University  Applied  Physics  Laboratory 
o  7393  -  Systems  Engineering  Approach  to  Total  Vehicle  Design  and  Integration:  Mr.  Walter  J.  Budd,  BAE  Systems 

Session  3D5 

o  7228  -  Total  System  Modeling:  A  System  Engineering  Application  of  the  Higraph  Formalism:  Mr.  Kevin  Fogarty,  SAIC 
©  7077  -  Near-field  RCS  and  Fuze  Modeling  and  Simulation:  Mr.  David  Hall,  Survice  Engineering  Company 
o  7174  -  Virtual  Battlespace  Center  for  Systems  Engineering:  Mr.  James  Hollenbach,  Simulation  Strategies,  Inc. 

MISSION  III:  NET  CENTRIC  OPERATIONS 
Session  3A6 

o  6954  -  SOAs  and  Net-Centric  Warfare-Similarities,  Differences  and  Conflicts:  Mr.  James  A.  Mazzei,  The  Aerospace  Corporation 
°  7374  -  Capitalizing  in  Migrating  Web  Service  Environments:  Mr.  Brian  Eleazer,  South  Carolina  Research  Authority 

Session  3B6 

o  6972  -  A  System  Engineering  Approach  to  Develop  a  Service-Oriented  Perspective:  Mr.  Rob  Byrd,  SI  International 
o  7413  -  Systems  Engineering  Approach  for  Assessing  a  Warfighter’s  Cognitive  Performance:  Mr.  James  Buxton,  U.S.  Army 

Session  3C6 

o  7105  -  Building  Net-Ready  Information  Interoperability  Performance  Indicator  Widgets  For  DoDAF  2.0  Dashboards:  Mr.  William  B.  Anderson, 
Software  Engineering  Institute 

°  7088  -  The  Benefit  of  Collaboration:  Integration  between  the  DoDAF  and  Systems  Engineering  Communities:  Mr.  Tim  Tritsch,  Vitech  Corporation 
o  7337  -  Modeling  Cognition  in  the  DoD  Architecture  Framework  for  Early  Concept  Development:  Dr.  John  M.  Colombi,  Air  Force  Institute  of 
Technology 

o  7046  -  Survivable  Network  Design  Framework,  Mr.  Dennis  Moen,  Lockheed  Martin 

o  7377  -  Joint  Surface  Warfare  Joint  Capability  Technology  Demonstration  -  Maturing  Weapon  Data  Link  Concepts  into  Operational  Capability,  Mr. 
Robert  Finlay  son,  John  Hopkins  University 

PALM  I:  REQUIREMENTS  DEVELOPMENT  &  MANAGEMENT 
Session  3A7 

o  7047-Stop  the  Pain:  Take  Some  Requirements  Definition  and  Management  for  Project  Success:  Mr.  Scott  Derby,  AVISTA  Incorporated 
o  7068-Daily  Challenges  in  Requirements  Engineering:  Mr.  Frank  J.  Salvatore,  High  Performance  Technologies,  Inc. 
o  7593-  Correlation  of  Types  of  Requirements  to  Verification  Methods:  Dr.  William  G.  Bail,  The  MITRE  Corporation 

Session  3B7 

°  7548-  Mission  Analysis  and  its  Impact  on  SE  Fundamentals:  Mr.  John  T.  McDonald,  Raytheon 
o  7055-  How  to  Write  ‘Lean  and  Mean’  Requirements:  Mr.  Tim  Olson,  Lean  Solutions  Institute,  Inc. 

PALM  I:  LOGISTRICS,  SUPPORTABILITY  &  SUSTAINMENT 
Session  3C7 

o  7180-A  Continuous  Process  View  of  Systems  Engineering  for  the  Sustainment  Phase:  Mr.  Paul  d.  Ratke,  OC  -  ALC 
o  7183-  Progress  Toward  the  Development  of  a  Reliability  Investment  Cost  Estimating  Relationship:  Mr.  Andy  Long,  LMI 
°  7235-  Future  Combat  Systems  (FCS)  Logistics  Systems:  Ms.  Soo  R.  Yoon,  Boeing 

Session  3D7 

°  7390  -  Systems  Engineering  of  Deployed  Systems:  Mr.  Robert  K.  Finlayson,  Johns  Hopkins  University,  Applied  Physics  Laboratory 
o  7383  -  Extending  Enterprise  Systems  for  an  Integrated  Logistics  Management  Environment:  Mr.  Mike  Korzenowski,  General  Dynamics  Land  Systems 
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7455-  The  Seven  Affordability  Sins  of  Logistics  System  Integration:  Dr.  Thomas  E.  Herald,  Lockheed  Martin 

PALM  II:  SOFTWARE 
Session  3A8 

o  7114-  Building  the  Next  Generation  of  Software  Engineers  -  Benchmarking  Graduate  Education:  Dr.  Arthur  Pyster,  Stevens  Institute  of  Technology 
o  7135  -  Improving  Work  Breakdown  Structure  (WBS)  Guidance  for  Weapons  Systems  with  Substantial  Software  Content:  Mr.  Christopher  Miller, 
OUSD/SE/SSA 

o  7232  -  ASN  (RD&A)  Initiatives  to  Improve  Integration  of  Software  Engineering  into  Defense  Acquisition  Related  Systems  Engineering:  Dr.  John  F. 
Miller,  The  MITRE  Corporation 

Session  3B8 

o  7198-  Software  Reuse  Readiness  Levels:  A  Framework  for  Decision  Making:  Mr.  Steven  Wong,  Northrop  Grumman 
o  7195  -  Counting  Software  Size:  Is  it  as  easy  as  Busying  a  Gallon  of  Gas?:  Ms.  Lori  Vaughan.  Northrop  Grumman 

PAM  II:  ARCHITECTURE 
Session  3C8 

°  7136-  Architecture  Trade-off  Analysis  Method®  (AT AM®)  for  System  Architecture  Evaluation:  Mr.  Michael  Gagliardi,  Software  Engineering  Institute 
o  7243  -  Method  for  Aligning  Architecture  Frameworks  and  System  Requirements:  Mr.  Richard  L.  Eilers,  IBM 

Session  3D8 

■  7428-  Adaptable  Architecture  for  System  of  Systems:  Mr.  Bruce  Schneider,  Applied  Physics  Lab  Johns  Hopkins  University 

■  7285  -  Universal  Architecture  Description  Framework:  Mr.  Jeffrey  O.  Grady,  JOG  System  Engineering 

■  7109  -  Applying  Open  Architecture  Concepts  to  Mission  and  Ship  Systems:  Mr.  John  M.  Green,  Naval  Postgraduate  School 

■  7273  -  US  Air  Force  Global  Persistent  Attack  Architecture,  Process,  &  Risk  Analysis:  Maj  Jeffrey  D.  Havlicek,  Air  Force  Center  for  Systems 
Engineering 


THURSDAY.  23  OCTOBER  2008 


BAYVIEW  III:  SYSTEMS  ENGINEERING  EFFECTIVENESS 
Session  4A1 

■  7697  -  Enhancing  Systems  Engineering  in  the  Department  of  Defense:  Mr.  Ceasar  Sharper,  ODUSD  /SSE 

■  7186  -  Air  Force  Implementation  of  NRC  “Pre-A  SE”  Study  Committee  Recommendations:  Mr.  Jeff  Loren,  AF/AQRE 

■  7281 -A  Holistic  Approach  to  System  Development:  Mr.  Douglas  T.  Wong,  NASA  Johnson  Space  Center 

Session  4B1 

■  7004  -  Operational  Concepts:  Mr.  James  R.  van  Gaasbeek,  Northrop  Grumman 

■  7296  -  The  Dangers  of  Oversimplifying  Availability:  Dr.  Jeffrey  M.  Harris,  General  Dynamics 

■  7214-Developing  and  Maintaining  the  Technical  Baseline:  Mr.  Michael  G.  Ucchino,  Air  Force  Institute  of  Technology 

Session  4C1 

■  7289  -  Process  Tailoring  Patterns  and  Frameworks  for  Accelerating  Systems  Engineering  Processes:  Mr.  Larry  J.  Earnest,  Northrop  Grumman 

■  7054  -  Using  Lean  Principles  and  Process  Models  to  Achieve  Measurable  Results:  Mr.  Tim  Olson,  Lean  Solutions  Institute,  Inc. 

■  7265-  Rocket  Motor  Development  Cycle  Time  -  Business  Process  Review:  Mr.  Jose  Gonzalez,  OUSD/PSA/LW&M 

BAYVIEW  II:  BEST  PRACTICES  &  STANDARDIZATION 
Session  4A2 

■  7076  -  Systems  and  Software  Life  Cycle  Process  Standards:  Foundation  for  Integrated  Systems  and  Software  Engineering:  Ms.  Teresa  Doran, 
TECHSOFT 

■  71 1 1  -  Improving  Process  Utilizations  with  Tools:  Mr.  Frank  J.  Salvatore,  High  Performance  Technologies,  Inc. 

■  7179  -  Integration  of  Systems  and  Software  Engineering:  Implications  from  Standards  and  Models  Applied  to  DoDs’  Acquisition  Programs:  Mr. 
Donald  Gantzer,  ODUSD/SSE 

Session  4B2 

■  7325  -  Applying  CMMI  High  Maturity  Practices  and  Leveraging  LEAN  Six  Sigma:  Mrs.  Ann  Hennon,  BAE  Systems 

■  7422  -  NDIA  CMMI  Working  Group:  Status  and  Plans:  Mr.  Geoff  Draper,  Harris  Corporation 

■  7441  -  Process  Enrichment  Boot  Camp,  Mr.  Victor  Elias,  High  Performance  Technology,  Inc 

■  7446  -  Best  Practices  Clearinghouse:  Making  Lessons  Learned  Come  Alive  and  Be  Practical,  Mr.  Forrest  Shull,  Fraunhofer  Center,  Maryland 

MISSION  II:  EDUCATION  &  TRAINING 
Session  4A5 

■  6944  -  Establishing  the  Need  for  Functional  Analysis  in  Systems  Development:  Dr.  Robert  J.  Monson,  Lockheed  Martin 

■  6946  -  Improving  Systems  Engineering  Execution  and  Knowledge  Management:  Mr.  Steven  C.  Head,  Boeing 

Session  4B5 
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■  7094  -  Development  and  Validation  of  a  Systems  Engineering  Competency  Model:  Dr.  Don  Gelosh,  SAIC 

■  7098  -  Accelerate  Performance  Improvements:  Systems  Engineering  Skills  Competency  Analysis  and  Training  Program  Development:  Mr. 

Steven  A.  Diebold,  General  Dynamics, 

■  7130  -  Concept  Defmiti-  A  Historical  Perspective:  Dr.  David  R.  Jacques,  Air  Force  Institute  of  Technology 

MISSION  III:  ENTERPRISE  HEALTH  MANAGEMENT 
Session  4A6 

■  7580  -  Engineering  Solutions  for  Fleet  Readiness  Centers  utilizing  an  Avionics  Rapid  Action  Team  Innovation  Cell:  Mr.  Bill  Birurakis,  PIDESO 

■  7447  -  Prognostics  as  an  Approach  to  Improve  Mission  Readiness  and  Availability:  Mr.  Sony  Mathew,  Center  for  Advance  Life  Cycle 
Engineering 

■  7613  -  Prognostics  Based  Health  Assessment  System  Approaches:  Mr.  Ronald  D.  Newman,  VSE  Corporation 

Session  4B6 

■  7520  -  NDIA  ID  Electronic  Prognostics  (E-Prog)  Task  Follow-on  Study  to  Quantify  Weapon  System  Benefits:  Mr.  Paul  Howard,  Paul  L.  Howard 
Enterprises 

■  7597  -  Enterprise  Health  Management  Emerging  Technology  Transition  Enabling  Plan:  Mr.  Chris  H.  Reisig,  Boeing 

LRU  Prognostics  Demonstration  Video  MPEG  Video  RealPlayer 

PALM  I:  LOGISTICS,  SUPPORTABILITY  &  SUSTAINMENT 
Session  4A7 

■  7481-  Defining  the  Prognostics  Health  Management  Enterprise  Architecture:  Mr.  Ethan  Xu,  Raytheon 

■  7131-  Sustaining  Systems  Engineering  -  The  A- 10  Example:  Dr.  David  R.  Jacques,  Air  Force  Institute  of  Technology 

■  7188-  Reliability  Centered  Maintenance  Applied  to  the  CH-47  Chinook  Helicopter-Universal  Principles  that  go  beyond  Equipment  Maintenance: 
Ms.  Nancy  Regan,  The  Force,  Inc. 

Session  4B7 

■  7207-  Sustainment  Engineering  versus  Systems  Engineering,  Is  There  A  Difference?:  Ms.  Karen  B.  Bausman,  AF  Center  for  Systems 
Engineering 

■  7064-  Reliability  Growth  Analysis  of  Mobile  Gun  System  during  PVT:  Dr.  Dmitry  Tananko,  GDLS 

PALM  II:  ARCHITECTURE 
Session  4A8 

■  7401-  Enabling  Systems  Engineering  with  an  Integrated  Approach  to  Knowledge  Discovery  and  Architecture  Framework:  Mr.  Michael  R. 
Collins,  Advantage  Development,  Inc. 

■  7453  -  Open  Architecture  in  Electronics  Systems:  Mr.  Bruce  R.  Bardell,  BAE  Systems 

■  7069  -  The  Value  of  Architecture:  Mr.  Frank  J.  Salvatore,  High  Performance  Technology,  Inc. 

Session  4B8 

■  7365  -  Enabling  the  Successful  Transition  from  Architecture  to  Concept  Design:  Mr.  Chris  Ryder,  Johns  Hopkins  University  Applied 
Physics  Laboratory 

■  7079  -  The  Benefits  of  Synergizing  Naval  Open  Architecture  Practices  and  Principles  with  Systems  Engineering  Processes:  Mr.  Mike 
Dettman,  PEO  C4I  -  NAVSEA 

■  7029  -  Concurrent  Increment  Sequencing  and  Synchronization  with  Design  Structure  Matrices  in  Software-Intensive  System  Development: 
Dr.  Peter  Hantos,  The  Aerospace  Corporation 
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CONFERENCE  OBJECTIVES 

This  conference  seeks  to  create  an  interactive  forum  for  Program  Managers,  Systems 
Engineers,  Software  Engineers,  Chief  Scientists,  and  Engineers  and  Managers  from 
government,  industry,  and  the  academic  communities  whose  interests  converge  on 
Defense  acquisition,  from  capabilities  analysis  through  operations  and  disposal.  This 
conference  will  provide  the  opportunity  to  learn  from  ones  peers  on  latest  techniques 
and  methodologies,  and  help  shape  policy  and  guidance  through  the  exchange  of 
innovative  procedures  and  lessons  learned  to  address  the  following  current  issues: 

•Effectiveness  of  Systems  Engineering 
•Program  Management 
•Architectures 

•Requirements  Development  &  Management 
•Interoperability  &  Systems  Integration 
•Software  &  Software-intensive  Systems 
•Network  Centric  Operations 
•System-of-Systems  Engineering 
•Modeling  &  Simulation 
•Integrated  Risk  Management 
•Aging  Aircraft 

•Logistics  &  Supportability  including  Performance  Based  Logistics 
•Life  Cycle  Systems  Management 

•Improved  Cycle  Times  for  Design,  Manufacture,  &  Repair  Process 
•Sustainment  &  Upgrade  of  Legacy  Systems 

•Application  of  Government  &  Industry  “Best  Practices”  Tools,  Methodologies,  & 
Technologies 

•System  Safety  -  Environment,  Safety  &  Occupational  Health  &  Human  Systems 
Integration 

•Improved  Mission  Readiness  &  Systems  Availability 
•Enterprise  Health  management  &  Integrated  Diagnostics 
•Systems  Engineering  Training  &  Education 
•Capability  Maturity  Model  Integration  (CMMI) 

•Integrated  Systems  Engineering,  Test,  &  Supportability  Discipline 
•Application  of  DoD  Initiatives: 

-Performance  Based  Business  Environment 
-System  Safety 
-Open  Systems 

-Simulation  Based  Acquisition 
-COTS  Integration 


BACKGROUND 

The  Department  of  Defense 
has  been  undertaking  a  major 
transformation  of  our  military  capability 

over  the  past  few  years  in  response  to  the  new  world  environment 
and  unforeseen,  ever-changing  threats.  The  ability  to  effect  this 
transformation  can  only  be  realized  if  our  Defense  Systems — space, 
air,  land,  sea,  and  under  sea — can  effectively  satisfy  mission  area 
and  capability  requirements,  and  achieve  and  sustain  a  high  degree 
of  interoperability,  systems  integration,  readiness,  availability,  and 
systems  safety,  with  affordable  cost.  We  believe  that  the  greatest 
opportunity  to  achieve  these  objectives  for  new  and  legacy  systems 
is  through  strong  technical  management  embodied  in  systems 


engineering  methodologies  and  processes,  on  the  part  of 
both  industry  and  the  DoD,  in  not  only  the  technical 
arms  but  the  management  &  program  management 
arms.  Strong  emphasis  on  systems  engineering  across  the  full 
acquisition  life  cycle,  from  concept  development  &  refinement 
through  deployment  &  sustainment,  is  a  key  enabler  of  improved 
performance  in  the  overall  acquisition  process  and  effectiveness. 
The  Systems  Engineering  Conference  is  an  annual  event  targeted  at 
exploring  the  role  of  technical  planning  and  execution  in  Defense 
programs  and  systems  from  a  variety  of  perspectives,  academic 
and  pragmatic,  by  the  entire  Defense  systems  engineering 
community. 


SYSTEMS  ENGINEERING  CONFERENCE 
GENERAL  INFORMATION 


GENERAL  INFORMATION 


CONFERENCE  ATTIRE 

Appropriate  dress  for  this  conference  is  business  casual  for  civilians  and  class  B 
uniform  for  military. 

During  conference  registration  and  check-in,  each  participant  will  be  issued  an 
identification  badge.  Please  be  prepared  to  present  a  picture  ID.  Badges  must  be 
worn  at  all  conference  functions. 


CONFERENCE  PROCEEDINGS 

Proceedings  will  be  available  on  the  web  through  the  Defense  Technical  Information 
Center  (DTIC),  and  will  be  available  one  to  two  weeks  after  the  conference.  You 
will  receive  notification  via  e-mail  once  proceedings  are  posted  and  available  on  the 
web. 


OTHER  INFORMATION 

Conference  Chair:  Mr.  Bob  Rassa,  Raytheon 

Conference  Technical  Program  Co-Chairs:  Dr.  Thomas  Christian,  USAF, 
Technical  Advisor,  Systems  Engineering,  USAF  AFMC/ASC;  Mr.  Steve  Henry, 
Northrop  Grumman 

Plenary:  Ms.  Kristen  Baldwin,  OSD/SSE 

Systems  Engineering  Effectiveness:  Mr.  A1  Brown,  Boeing;  Ms.  Sharon  Vannucci,  OSD 

Logistics  Supportability  &  Sustainment:  Mr.  Joel  Moorvich,  Raytheon 

Involving  Test  &  Evaluation  in  SE:  John  Lohse,  Raytheon;  Darlene  Mosser-Kerner,  OSD 

Program  Management:  Mr.  Hal  Wilson,  Northrop  Grumman 

Modeling  &  Simulation:  Mr.  Jim  Hollenbach,  SIMSTRAT,  Inc.;  Mr.  Gary  Belie, 
Lockheed  Martin 

Net  Centric  Operations:  Mr.  Jack  Zavin,  ASD(NII);  Dr.  Rich  Eilers,  IBM 
Best  Practices  &  Standardization:  To  be  announced 
Software:  Mr.  Paul  Croll,  CSC 

Education  &  Training  in  SE:  Mr.  Mike  Ucchino,  USAF/AFIT/CSE 

Enterprise  Health  Management:  Mr.  Dennis  Hecht,  Boeing;  Mr.  Howard  Savage, 
Savage  Consulting 

System  Safety,  ESOH  &  HIS:  Mr.  Sherman  Forbes,  USAF;  Ms.  Paige  Ripani, 
Booz  Allen  Hamilton 

Requirements  Development  &  Management:  Mr.  Bob  Scheurer,  Boeing 
Architecture:  Mr.  Joe  Kuncel,  Northrop  Grumman;  Mr.  John  Palmer,  Boeing 
Practical  SE  Experience:  To  be  Announced 
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CONFERENCE  AGENDA 


SUNDAY,  OCTOBER  19, 2008 

5:00  pm  -  7:00  pm 

Registration  for  Tutorials  and  General  Conference 
(Tutorials  are  an  additional  $250.00  registration  fee) 

MONDAY,  OCTOBER  20, 2008 

7:00  am  -  5:00  pm 

7:00  am  -  8:00  am 

Registration 

Continental  Breakfast  for  Tutorial  Attendees  ONLY 

8:00  am  -  12:00  pm 

(Tutorials  are  an  additional  $250.00  registration  fee) 

Tutorial  Tracks 

(Please  refer  to  the  following  pages  for  Tutorial  Schedule) 

12:00  pm  -  1:00  pm 

1:00  pm  -  5:00  pm 

5:00  pm  -  6:00  pm 

Lunch  for  Tutorial  Attendees  ONLY 

Tutorial  Tracks  Continued 

Reception  in  the  Regency  Annex  (Open  to  All  Participants) 

TUESDAY,  OCTOBER  21, 2008 

7:15  am  -  5:00  pm 

7:15  am  -  8:15  am 

8:15  am  -  8:30  am 

Registration 

Continental  Breakfast 

Introductions  &  Opening  Remarks: 

Mr.  Sam  Campagna,  Directory  Operations,  NDIA\ 

Mr.  Bob  Rassa,  Director,  Systems  Supportability,  Raytheon ;  Chair,  Systems  Engineering  Division 

8:30  am  -  9:45  am 

Keynote  Addresses: 

HON  Charles  McQueary,  Director,  Operational  Test  &  Evaluation; 

Gen  Les  Lyles,  USAF  (Ret) 

9:45  am  -  10:15  am 

10:15  am  -  12:15  pm 

Break 

Plenary  Session:  Executive  Panel 

Moderator: 

Ms.  Kristin  Baldwin,  Deputy  Director,  Software  Engineering  &  System  Assurance 

Panelists: 

12:15  pm  -  1:30  pm 

Mr.  Terry  Jaggers,  Director,  SAF/AQR  (Science,  Technology  &  Engineering) 

Mr.  Carl  Siel,  Chief  Systems  Engineer,  ASN(RDA)  CHENG 

Mr.  Kelly  Miller,  Director,  Systems  Engineering,  NSA 

Mr.  Ross  Guckert,  Assistant  Deputy,  Acquisition  &  Systems  Integration  ASA(ALT) 
Luncheon  with  Speaker  in  the  Regatta  Pavilion 

Dr.  Ronald  Jost,  Deputy  Assistant  Secretary  of  Defense,  C3,  Space  &  Spectrum 

1:30  pm  -  5:15  pm 

Concurrent  Sessions 

(Please  refer  to  the  following  pages  for  session  schedule) 

5:15  pm  -  6:30  pm 

Reception  in  the  Regatta  Pavilion 

SYSTEMS  ENGINEERING  CONFERENCE 
PROGRAM  AGENDA 


CONFERENCE  AGENDA,  CONTINUED 

WEDNESDAY,  OCTOBER  22, 2008 

7:00  am  -  5:00  pm  Registration 


7:00  am  -  8:00  am 

8:00  am  -  12:00  pm 

Continental  Breakfast 

Concurrent  Sessions 

(Please  refer  to  the  following  pages  for  session  schedule) 

12:00  pm  -  1:30  pm 

Luncheon  with  Speaker  in  the  Regatta  Pavilion 

Ms.  Shannon  Cunniff,  Director,  Emerging  Containments:  Office  of  Under  Secretary  of  Defense 
(Installations  and  Environment) 

1:30  pm  -  5:15  pm 

Concurrent  Sessions 

(Please  refer  to  the  following  pages  for  session  schedule) 

THURSDAY,  OCTOBER  23, 2008 

7:00  am  -  3:00  pm 

7:00  am  -  8:00  am 

8:00  am  -  12:00  pm 

Registration 

Continental  Breakfast 

Concurrent  Sessions 

(Please  refer  to  the  following  pages  for  session  schedule) 

12:00  pm  -  1:00  pm 

1:00  pm  -  3:00  pm 

Awards  Lunch  in  the  Regatta  Pavilion 

Concurrent  Sessions 

(Please  refer  to  the  following  pages  for  session  schedule) 

3:00  pm 

Conference  Adjourns 

Tutorial  Sessions  -  Monday,  October  20, 2008 
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Thursday,  October  23, 2008 


7214-Developing  and  Maintain¬ 
ing  the  Technical  Baseline 

Mr.  Michael  G.  Ucchino,  Air 

Force  Institute  of  Technology 

7422  -  NDIA  CMMI  Working 

Group:  Status  and  Plans 

Mr,  Geoff  Draper,  Harris 

Corporation 

7255-  Integrated  Change 

Control  for  the  Concurrently 

Developed  Complex  Systems 

-  Lessons  Learned 

Mr.  Alexander  J.  Polack,  The 

Aerospace  Corporation 

7278  -  Integrating  Metics  with 

Qualitative  Temporal  Reasoning 

for  Constraint-Based  Expert 

Systems 

Dr.  James  A.  Crowder,  Raytheon 

7130  -  Concept  Definiti-  A 

Historical  Perspective 

Dr.  David  R.  Jacques,  Air  Force 

Institute  of  Technology 

7029  -  Concurrent  Increment 

Sequencing  and  Synchronization 

with  Design  Structure  Matrices 

in  Software-Intensive  System 

Development 

Dr.  Peter  Hantos,  The  Aerospace 

Coporation 

7296  -  The  Dangers  of 
Oversimplifying  Availability 

Dr.  JeJfrey  M.  Harris,  General 
Dynamics 

7400  -  Systems  Engineering 
Initiative  -  How  do  you 
Implement  a  New  Lessons 

Learned  Process  and  Tool  on  a 
Legacy  Program? 

Mr.  Ray  A.  Polo,  Boeing 

74 59  -  Multi-Factor  Risk 
Management 

Ms.  Laura  West,  BAE  Systems 

7102  -  Reengineering  Electronic 

Warefare:  Shifting  From  Platform  - 

To  Capability  -  Centric  Engineering 

Mr.  William  B.  Anderson,  Software 

Engineering  Institute 

7098  -  Accelerate  Performance 

Improvements:  Systems 

Engineering  Skills  Competency 

Analysis  and  Training  Program 

Development 

Mr.  Steven  A.  Diebold,  General 

Dynamics, 

7597  -  Enterprise  Health  Man¬ 

agement  Emerging  Technology 
Transition  Enabling  Plan 

Mr.  Chris  H.  Reisig,  Boeing 

7064-  Reliability  Growth 

Analysis  of  Mobile  Gun  System 

during  PVT 

Dr.  Dmitry  Tananko,  GDLS 

7079  -  The  Benefits  of  Synergizing 

Naval  Open  Architecture  Practices 

and  Principles  with  Systems 

Engineering  Processes 

Mr.  Mike  Dettman,  PEO  C4I 

-  NAVSEA 

7004  -  Operational  Concepts 

Mr.  James  R.  van  Gaasbeek, 
Northrop  Grumman 

7325  -  Applying  CMMI 

High  Maturity  Practices  and 
Leveraging  LEAN  Six  Sigma 

Mrs.  Ann  Hennon,  BAE  Systems 

7363  -  Integrated  Risk  and  Op¬ 
portunity  Management 

Ms.  Audrey  Dorofee,  Software 
Engineering  Institute 

7063  -  Product  Platforms  in 
Support  of  Rapid  Response 
to  DOD  In-Theatre  Force 
Protection  Needs 

Dr.  Steven  B.  Shooter,  Bucknell 
University 

7094  -  Development  and 

Validation  of  a  Systems 

Engineering  Competency  Model 

Dr.  Don  Gelosh,  SAIC 

7520  -  NDIA  ID  Electronic 

Prognostics  (E-Prog)  Task 

Follow-on  Study  to  Quantify 

Weapon  System  Benefits 

Mr.  Paul  Howard,  Paul  L. 

Howard  Enterprises 

7207-  Sustainment  Engineering 

versus  Systems  Engineering,  Is 

There  A  Difference? 

Ms.  Karen  B.  Bausman,  AF 

Center  for  Systems  Engineering 

7365  -  Enabling  the  Successful 

Transition  from  Architecture  to 

Concept  Design 

Mr.  Chris  Ryder,  Johns  Hopkins 

University  Applied  Physics  Laboratory 

Bayview  III 

Systems  Engineering 
Effectiveness 

Session  4B1 

Bayview  II 

Best  Practices  & 
Standardization 

Session  4B2 

Bayview  1 

Program 

Management 

Session  4B3 

Mission  1 

Practical  Systems 
Engineering 

Experience 

Session  4B4 

Mission  II 

Education  &  Training 
Session  4B5 

Mission  III 

Enterprise  Health 
Management 

Session  4B6 

Palm  1 

Logistics,  Support- 
ability  &  Sustainment 
Session  4B7 

Palm  II 

Architecture 

Session  4B8 

X9uuv  Aou860y  014  u|  >|B0jg 


SYSTEMS  ENGINEERING  CONFERENCE 
TRACK  SESSIONS 


r 

Thursday,  October  23, 2008 


1:30  pm  -  3:00  pm 


Bayview  III 

Systems  Engineering 
Effectiveness 

Session  4C1 

7289  -  Process  Tailoring 

Patterns  and  Frameworks 
for  Accelerating  Systems 
Engineering  Processes 

Mr.  Larry  J.  Earnest,  Northrop 
Grumman 

7 054  -  Using  Lean  Principles 
and  Process  Models  to  Achieve 
Measurable  Results 

Mr.  Tim  Olson,  Lean  Solutions 
Institute,  Inc. 

7265-  Rocket  Motor 

Development  Cycle  Time 
-  Business  Process  Review 

Mr.  Jose  Gonzalez,  O  USD/PSA/ 
LW&M 

Bayview  II 

Best  Practices  & 
Standardization 

Session  4C2 

7441  -  Process  Enrichment  Boot 
Camp  -  An  Intensive  Introduction  to 
a  Generic,  Enterprise-wide,  Strategic 
Communication  and  Continuous 
Improvement  Methodology 

7446-  Making  Lessons 

Learned  Come  Alive  and  be 

Practical 

Mr.  Victor  Elias,  High  Performance 
Technologies  Inc. 

Mr.  Forest  Shull,  Fraunhofer 

Center  Maryland 

Bayview  1 

Program 

Management 

Session  4C3 

7067-  Estimating  Systems 
Engineering  Level  Of  Effort 

Mr.  Frank  Salvatore,  High 
Performance  Technologies,  Inc. 

7189-  The  Integrated  Natural 
Environment  Authoritative 
Representation  Process 
(INEARP)  and  Beyond 

Maj  James  Everitt,  Air  &  Space 
Natural  Environment  M&S 
Executive  Agent 

Mission  1 

Practical  SE 

Experience 

Session  4C4 

7417  -  VIRGINIA 
(SSN-774)  Class 
Systems  Engineering  to 
Reduce  Total  Ownership 
Cost 

7463  -  The  C-17  PIO 
Team 

7497-  Accuracy  Control 
Tools,  Technology, 
and  Processes  used  for 
Addressing  Hull  Fairness 

Mr.  Steve  Lose,  Naval  Sea 
Systems  Command 

Mr.  David  Murray,  Boeing 

Mr.  Stephan  H .  Hankins, 
Northrop  Grumman 

Mission  II 

Education  & 

Training 

Session  4C5 

7308  -  PeaceKeeper 
Intercontinental  Ballistic  Missile 
Systems  Engineering  Case 

Study 

Mr.  Charles  M.  Garland, 

Air  Force  Center  for  Systems 
Engineering 

7474  -  CAPTURE  of  Critical 
Engineering  Skills  and  Knowledge 

Mrs.  Ann  Hennon,  BAE  Systems, 

Promotional  Partner 


Thank  You  to  Our 
Promotional  Partner! 


LOCKHEED 


LOCKHEED  MARTIN  CORPORATION 

Lockheed  Martin  is  a  premier  systems  integrator  and  global  security  enterprise  principally  engaged  in  the  research, 
design,  development,  manufacture,  integration  and  sustainment  of  advanced  technology  systems,  products  and  services. 
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Performance  Results  of  CMMI  -  Based  Process  Improvement 
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Why  results  vary  - 1 


Two  different  approaches  to  CMMI  based  Process  Improvement: 

•  Bureaucratic  improvement  that  comes  to  life  only  when  assessments 
are  to  be  performed 

•  Improvement  efforts  that  are  based  on  achieving  business  objectives 
which  are  embedded  into  the  culture  of  an  organization  and  actively 
supported  by  the  entire  staff 
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Why  results  vary  -  2 
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Bureaucratic  Improvement 


Bureaucratic  Improvements  can  be  very  successful  in  changing  the 
organizational  culture.  However  it  doesn’t  fundamentally  change  the 
developers  individual  behavior  or  processes. 

Resulting  in  continued  quality,  cost  and  schedule  issues.  Because 
ultimately  only  the  developers  can  control  the  quality  of  the  product, 
which  directly  impacts  the  cost  and  schedule. 


Process  qa 


CM  V&V 


How  to  get  the  performance  you  expect  using  CMMI 


Improvement  efforts  that  are  based  on  achieving  business 
objectives  which  are  embedded  into  the  culture  of  an 
organization  and  actively  supported  by  the  entire  staff: 

Achieving  a  maturity  rating  doesn’t  guarantee  improved 
performance 

To  get  high  performance,  you  need 
to  build  a  solid  foundation  from  the 
beginning 

Performance  becomes  an  enabler 
for  high  maturity 
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V&V 

Process 

MA 

PP 

QA 

CM 
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Definitions 


High  Performance  - 

High  performance  means  obtaining  superior 
outcomes. 


High  Maturity - 

Implementing  the  concepts  and  practices  at 
levels  4  and  5  of  CMMI. 


High  Maturity  Practices  - 

The  "specific  practices"  and  "generic  practices"  at  levels  4  and  5  of  CMMI. 


=-  Software  Engineering  Institute  CarnegieMellon 


Why  CMMI  isn’t  enough 
©2008  Carnegie  Mellon  University 


7 


Align  Business  Objectives 


Are  we  getting  more  business  moving  to  a  higher  maturity? 
Are  we  shipping  (releasing)  higher  quality  products? 

Do  we  have  better  performance? 

Do  our  products  have  more  functionality? 

Are  we  reducing  our  costs? 

Are  we  meeting  our  schedules? 


How  do  we  get  high  performance  from  high  maturity? 
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Prerequisites  for  High  Performance 


Before  an  organization  can  perform  high  maturity  activities,  it  must: 

-  Gather  and  use  data  at  all  organizational  levels 

-  Defined  operational  processes  that  specify  how  and  when  the  data  are 
gathered 

-  Faithfully  execute  the  defined  processes 


This  implies  that  individuals  and  teams  gather  data  on  their  own  and  use 
the  data  to  plan  and  perform  work 
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To  Get  High  Performance,  Address  Team  and 
Individual  Discipline 


A  high-performing  organization 
must  be  built  of  high  performing 
teams. 

High  performing  teams  must  be 
built  of  high-performing  individuals. 

High-performing  individuals 
must  be  disciplined  to  gather 
and  use  their  own  data. 
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For  a  successful  case  study  showing  the  integration  of  CMMI  and  TSP,  please  see  — G/IMI  Level  5  and 
the  Team  Software  Process”  by  Webb,  Miluk,  and  Van  Buren  in  CrossTalk  April  2007. 
http://www.stsc.hill.af.mil/crosstalk/2007/04/0704WebbMilukVanBuren.html 
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Operationalizing  CMMI  Practices 


What  does  operationalize  mean? 

•  To  put  something  to  use 

What  are  characteristics  of  an  -operationalized”  process? 

•  The  people  who  use  the  process  own  the  process  and 
have  the  authority  to  adapt  and  improve  it. 

•  The  — process  owers”  are  in  the  best  position  to 
understand  the  process  strengths  and  weaknesses. 

•  If  people  — owithe  process,”  they  will  be  more  willing  to 
fairly  evaluate  process  changes. 
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Once  you  collect  data,  what  do  you  do  with  it? 


Discussion: 

•  Why  do  you  need  to  periodically  review  your 
process  data? 

•  How  often  should  you  review  your  process  data? 

•  What  happens  if  you  review  your  process  data  too 
often?  too  seldom? 


If  you  have  already  set  goals,  you  start  by 
understanding  your  performance  against  those 
goals. 
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Analyzing  Performance 

Analyze  your  performance  with  respect  to  size 
estimation,  effort  estimation,  and  quality  management  to: 

•  understand  your  current  performance 

•  identify  your  highest-priority  areas  for  improvement 

•  establish  challenging  but  achievable  goals,  and 

•  define  corresponding  improvement  actions  to  meet  those  goals 

•  define  actions  to  address  challenges  you  will  face  in  making  those 
changes 
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Analysis  of  Size 

Review  your  performance  on  size  estimating  accuracy.  For 
example: 

•  How  much  did  your  size  estimating  accuracy  change?  Why? 

•  Do  I  have  a  tendency  to  add/miss  entire  parts? 

•  Do  I  have  a  tendency  to  misjudge  the  relative  size  of  parts? 

•  Do  I  need  to  calculate  relative  size  range  data  using  my  historical 
data? 

•  Based  on  my  historical  size-estimating  accuracy  data,  what  is  a 
realistic  size-estimating  goal  for  me? 

•  How  can  I  change  my  process  to  meet  that  goal? 


Estimating  Accuracy 
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Analysis  of  Time  Estimating  Accuracy 

Review  your  performance  on  effort  estimating  accuracy.  For 
example: 

•  How  much  did  your  effort  estimating  accuracy  change?  Why? 

•  Is  my  productivity  stable?  Why  or  why  not? 

•  How  can  I  stabilize  my  productivity? 

•  How  much  are  my  time  estimates  affected  by  the  accuracy 
of  my  size  estimates?  (Would  multiple  regression  help  me?) 

•  Based  on  my  historical  time-estimating  accuracy  data, 
what  is  a  realistic  time-estimating  goal  for  me? 

•  How  can  I  change  my  process  to  meet  that  goal? 
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Defect  and  Yield  Analysis 


For  example: 

•  What  type  of  defects  do  I  inject  during  design  and  coding? 

•  What  trends  are  apparent  in  defects  per  size  unit  (e.g.,  KLOC)  found 
in  reviews,  compile,  and  test? 

•  What  trends  are  apparent  in  total  defects  per  size  unit? 

•  How  do  my  defect  removal  rates  (defects  removed/hour)  compare 
for  design  review,  code  review,  compile,  and  test? 

•  What  are  my  review  rates  for  design  review  and  code  review? 

•  What  are  my  defect-removal  leverages  for  design  review,  code 
review,  and  compile  versus  unit  test? 

•  Is  there  any  relationship  between  yield  and  review  rate  for  design 
and  code  reviews? 
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Leading  vs.  Lagging  Indicators 


F  Software  Engineering  Institute  Carnegie  Mellon 


Case  Study  - 1 


Cumulative  Earned  Value 


♦  Cumulative  Planned  Value 
— ■ — Cumulative  EV 

Cumulative  Predicted  Earned  Value 
— • —  Baseline  Cumulative  Plan  Value 
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Case  Study  -  2 


TSP  Week  Summary  -  Form  WEEK 

Name 

Team 

Consolidation 

Date 

11/8/2007 

Example  week 

Status  for  Week 

^  2 

HlHlUected  Assembly 

Cycle  1 

Week  Date 

7/2/2007  SYSTEM 

Plan  / 

Plan  - 

Task  Hours  %Change 

Weekly  Dajg^ 

_ Actual 

Actual 

Project  End  Dates 

Baseline 

1280.1 

Schedule  hours  for  this  (fieek 

45.5 

26.9 

^\1.69 

18.6 

Baseline  2/4/2008 

Current 

1332.1 

Schedule  hours  this  cycle  to 

^  86.9 

AMb 

1 .79 

38.3 

Plan  2/4/2008 

%Change 

4.1% 

Earned  value  for  this  week 

1.3 

0.7 

1.86 

0.6  ^"'Predicted  11/16/20$T 

Earned  value  this  cycle  to  date 

3.7 

3.4 

1.10 

0.3 

To-date  hours  for  tasks  completed 

44.7 

31.9 

1.40 

To-date  average  hours  per  week 

43.4 

24.3 

1.79 

EV  per  completed  task  hour  to  date 

0.075 

0.105 

A  team  is  in  week  2  of  7  month  plan. 

The  team  is  behind  10%  in  Earned  Value  but  the  projected  date  for  project  completion  is  2 
years  late —  what  is  the  problem? 

The  team  on  average  is  only  getting  a  little  more  than  half  of  their  planned  on-project  task 
hours. 

(1)  Understand  why  the  predicted  project  completion  is  two  years  late? 

(2)  Why  aren’t  team  members  achieving  planned  on-project  task  hours? 
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Case  Study  -  How  Do  You  Get  This  Information? 


From  having  operationally,  defined  processes  (e.g.,  development  process) 
From  basic,  measurement  data 

-  Operational  measures  (size,  effort,  schedule,  quality) 

-  Measurement  Definitions  (task  hour,  defect,  ...) 

From  tools 

-  To  record  and  analyze  data 
From  having  a  realistic  plan 

-  Developed  by  team  members  who  use  their  own  data  for  estimating 
and  planning 
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Operational  Definition 


Task  Hour 

•  Count  effort  applied  to  a  specific  project  task 

•  Do  not  count 

•  Break  time 

•  Project  tasks  not  in  the  earned  value  plan 

•  Non-project  tasks 
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Operational  Definition 


Earned  Value 

•  Planned  Value  for  task  =  estimated  effort  (cost)  for  task  divided  by  sum 
of  estimated  effort  for  all  project  tasks 


•  Earned  Value  credited  when  task  is  complete 


•  In  this  definition  Earned  value  always  approaches  1 .0  as  the  project 
nears  completion 
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Each  Week:  (Actual  -  Planned)  Effort  [hours] 


-  -Effort  (hr)  Act-Plan 
— mean 
— UCL 
— LCL 


The  team  addressed  the  project  effort  problem. 
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Variation 


Moving  Range 


Week 


-  -Moving  Range 

—  Linear  (Moving  Range) 
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Week  8,  Schedule  Progress  (Earned  Value) 


After  initially  falling  farther  behind,  weekly  progress  stabilizes. 
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Weekly  Status  Report 


Weekly  status  reviews: 

•  Plan  assumptions 

—  Effort  plan 
—  Upcoming  work  tasks 

•  Project  status 

—Actual  effort 
—  Earned  Value 
—  Cost  Performance 

•  Projections  based  on  status  and  history 
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Week  8  Team  Report 


TSP  Week  Summary  -  Form  WEEK 


Name 

Consolidation 

Date 

11/8/2007 

Team 

Example  week  8 

Status  for  Week 

▲ 

▼ 

8 

Selected  Assembly 

Cycle 

1 

Week  Date 

8/13/2007 

SYSTEM 

Plan  /  Plan  - 

Task  Hours  %Change 

Weekly  Data 

Plan  Actual  Actual  Actual 

Project  End  Dates 

Baseline 

1280.1 

Schedule  hours  for  this  week 

_ 17  °  -  'll!  1.08  3.4 

Baseline 

2/4/2008 

Current 

%Change 


1358.8 


6.1% 


Schedule  hours  this  cycle  to  dati 
Earned  value  for  this  week 
Earned  value  this  cycle  to  dati 
To-date  hours  for  tasks  completed 
To-date  average  hours  per  week 
EV  per  completed  task  hour  to  date 


0.074 


0.092 


The  team  actions  have  been  effective: 

•  Cumulative  hours  have  not  caught  up 

•  The  team  is  9%  ahead  of  schedule 

•  The  predicted  end  date  is  now  2  months  late  rather  than  2  years 
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Are  They  Following  Their  Process? 


Plan  and  Actual  Time  in  Phase 


Phase  Requirements  through  System  Test 


=-  Software  Engineering  Institute  Carnegie  Mellon 


Size  Estimation 
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Case  Study  Wrap-up 


Teams  and  individuals  need  to  assess  performance  with  respect  to  goals: 

•  Did  we  achieve  our  performance  goals?  Why  or  why  not? 

•  Where  do  we  need  to  improve?  What  could  we  do  differently? 

How  would  it  change  our  performance? 

•  What  kind  of  analyses  need  to  be  performed? 
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Summary 


Build  high  performance  through  teams 

Enable  high  maturity  capabilities  by  building  a  solid  foundation 

CMMI  and  TSP  are  mutually  reinforcing — 

•  CMMI  provides  the  principles  for  process  improvement  and 
organizational  focus 

•  TSP  can  be  useful  for  providing  team  discipline  and  operationalizing 
CMMI  practices 


Software  Engineering  Institute 
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Questions? 


TSP  SYMPOSIUM  2009 


Tim  Chick 

tchick@sei.cmu.edu 

412-268-1473 


PSP/TSP  website: 

http://www.sei.cmu.edu/tsp 


www.sei.cmu.edu/tsp/symposium.html 


4th  Annual  Software  Engineering  Institute 
Team  Software  Process  Symposium 


September  21-24, 2009  •  New  Orleans,  LA 


?  Software  Engineering  Institute 


Carnegie  Mellon 


Bourbon 
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NAVAIR  Benefits  from  TSP 


Program 

Size  of  Program 

Defect  Density 

(Defects/KSLOC)) 

Cost  Savings 
from  Reduced 
Defects 

AV  JMPS 

443  KSLOC 

0.59 

$2,177,169 

P-3C 

383  KSLOC 

0.6 

$1 ,478,243 

Program 

Schedule  Variance 

Cost  Variance 

AVJMPS 

0.5%  overrun 

1.5%  overrun 

H2.0 

1.1%  overrun 

6.9%  overrun 
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Quality  Benefits 


•  TSP  dramatically  reduces  the  effort 
and  schedule  for  system  test. 

Most  defects  are  removed  during 
reviews  and  inspections  at  a  cost  of 
2  to  25  minutes  per  defect. 

•  System  test  removal  costs  run  from 
to  2  to  20  hours  per  defect. 

•  These  benefits  continue  after 
delivery. 

•  lower  support  costs 

•  satisfied  customer 

•  better  resource 
utilization 


IpB-  Software  Engineering  Institute 


TSP  System  Test  Performance  Comparison  w/Table 


60%  ■ 
50%  ■ 
40%  ■ 
30%  ■ 
20%  ■ 
10%  ■ 
0%  ■ 

□  TSP  Min. 

□  TSP  Avg. 

■  TSP  Max. 

□  Typical  Projects 
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Reviews  and  Inspections  Save  Time 


Xerox  found  that  TSP  quality  management  practices  reduced  the  cost 
of  poor  quality  by  finding  and  removing  defects  earlier  when  costs  are 
lower. 


1600 

1400 

1200 


w  1000 
<D 


3 

C 


800 

600 

400 

200 

0 


Defect  Removal  Time  by  Phase 


1405 

5  22  2  25  32 

J _ _ l  l _ _ _ 1  1 _ _ 1  1 _ _ 

Design  Design  Code  Code  Unit  System 

Review  Inspect  Review  Inspect  Test  Test 

Removal  Phase 
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Intuit  Quality  Improvement 


•  TSP  reduced  defects  found  in  system  test  by  60%  over  the  previous 
two  releases  of  QuickBooks  2007  release. 

•  Intuit  has  also  recently  reported  a  savings  of  $20M  from  a  reduction  in 
customer  support  calls  on  QuickBooks  2007. 


Results  at  Intuit:  Improved  Quality 


Curnulatcve-  Delects  Found  — acr*  ' 


In  2007  ^60°/o  fewer  defects  were  found 
in  System  Test  than  the  previous  two  releases 
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Intuit  Productivity  Improvement 


•  By  putting  a  quality  product  into  system  test  Intuit  improved  productivity 
and  reduced  cost  while  delivering  33%  more  functionality  than  planned. 


Results  at  Intuit:  Productivity 


■  During  2007  over  60%  of  Intuit’s  Small  Business 
Division  used  TSP 

■  TSP  was  a  major  contributor  to  the  QuickBooks  2007 
release 

■  It  was  the  smoothest  release  anyone  can  remember: 

■  On  time  delivery  of  all  planned  scope 

■  13  new  features  were  added  during  the  cycle(33% 
of  initial  scope) 

■  Saved  $700K  in  temporary  testing  staff  expenses 

■  Level  of  automated  testing  coverage  was  doubled 
compared  to  previous  year 


Focused  improvements  helped  deliver  a  great  release 
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Improving  Task  Hours 


•  At  Allied  Signal  average  task  hours 
per  developer  per  week  were 
improved  from  9.6  hours  to  15.1 
hours  through  quiet  time,  process 
documentation,  more  efficient 
meetings,  etc. 

•  This  is  equivalent  to  a  57% 
increase  in  productivity. 

•  If  you  didn’t  have  such  detailed 
information,  would  you  even  know 
that  you  had  a  problem?  Or  an 
opportunity  for  such  dramatic 
improvement? 


s=r  Software  Engineering  Institute 


C< 


T ask  Hours 


Average  Task  Hours  Per  Week 


Actual  Task  Hours  per  Week 
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Intuit  Test  Schedule  Reduction 


•  From  data  on  over  40  TSP  teams,  Intuit  has  found  that 

•  post  code-complete  effort  is  8%  instead  of  33% 
of  the  project 

•  for  TSP  projects,  standard  test  times  are  cut 
from  4  months  to  1  week 

•  Testing  time  is  reduced  from  four  months  to  one  month. 


Non-TSP 


Development 


TSP 


Development 


Test 


Test 
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Microsoft  Schedule  Improvement 


•  First-time  TSP  projects  at  Microsoft  had  a  10  times  better  mean 
schedule  error  than  non-TSP  projects  at  Microsoft  as  reflected  in  the 
following  table. 


Microsoft  Schedule  Results 

Non-TSP  Projects 

TSP  Projects 

Released  on  Time 

42% 

66% 

Average  Days  Late 

25 

6 

Mean  Schedule  Error 

10% 

1% 

Sample  Size 

80 

15 
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Work-Life  Balance 


•  People  are  your  most  important  resource. 

Finding  and  retaining  good  people  is  critical  to  long-term  success. 

Intuit  found  that  TSP  improved  work-life  balance,  a  key  factor  in  job  satisfaction. 
Results  at  Intuit:  Improved  Work-Life  Balance 


■  Half  as  many  weekend  source  check-fns 
(<3%) 

■  Reduced  $  on  dinners  as  measured  by  PSS  - 
“Pizza  Slices  Served” 


TSP  helped  improved  employee  work  life  balance 
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Intuit  TSP  Survey  Results 


•  Improved  work-life  balance  with  TSP  is  reflected  in  job  satisfaction 
surveys. 


In  my  work  group,  we  continually 
improved  our  work  processes 

I  feel  encouraged  to  come  up  with  new 
and  better  ways  of  doing  things 

I  like  the  kind  of  work  I  do 


I  feel  proud  to  work  at  Intuit 


I  have  opportunities  to  improve  my  skills 


60  80  100 
%  Favorable  ■  TSP  ■  Non-TSP 


"Engineers  love  it...  Once  they  adopt  it  they  can't 

imagine  going  back" 


Source:  Intuit 
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Systems  Engineering  Conference 


Reduction  of  Total  Ownership  Costs 
(R-TOC)  and  Value  Engineering  (VE) 
in  the  Defense  System’s  Life  Cycle 


Mr.  Chet  Bracuto 

Office  of  the  Secretary  of  Defense 
Acquisition,  Technology  &  Logistics 
Systems  and  Software  Engineering 

Dr.  Danny  Reed 

Institute  for  Defense  Analyses 
October  22,  2008 
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R-TOC  Genesis 


•  Initiated  in  1999  by  the  USD(AT&L)  to  address: 

-  O&S  cost  growth  at  expense  of  force  modernization  and 
readiness 


-  O&S  budget  constraints  limit  programs  to  near-term,  critical 
solutions  only 

-  R-TOC  program  seeks  to  seed  O&S  cost  avoidance 
solutions  that  have  broader  impact 


-  Thirty  Pilot  Programs 


Rising 
O&S  Cost 


Less  $  for 
Modernization 


Aging 

Equipment 


Fixed  Top  Line 


Declining  Future 
Readiness 


DEATH  SPIRAL 


USD(AT&L)  FY  2005  R-TOC  Goal 


USD(AT&L)  Goal:  .  .reduce  the  O&S  cost  of  fielded 
systems  (excluding  manpower  and  fuel)  by  20% 

(compared  to  current  FY  1998  levels)  by  the  year  2005.” 

“Overall,  each  Service’s  O&S  reduction  plans  will  be  based 
on  tradeoffs  among  these  three  areas  for  savings: 

1 .  Reduced  demand  from  weapon  systems  via  reliability 
and  maintainability  improvements 

2.  Reduced  supply  chain  response  times,  leading  to 
reduced  spares,  system  support  footprint,  and  depot 
needs 

3.  Competitive  sourcing  of  product  support,  leading  to 
streamlining  and  overhead  reductions” 


FY  2005  O&S  Savings 


•  FY  2005  cost  avoidances  exceeded  $2.1  B 

•  Projected  life  cycle  cost  avoidances  will  exceed 
$76B,  for  the  R-TOC  Pilot  Programs 

O&S  Costs  Can  Be  Reduced!! 


Life  Cycle  Savings  Provides  a  Focus 
on  Long  Term  Benefits 


New  Strategic  Direction 


•  With  the  successful  completion  of  the  Pilot  Programs 
FY  2005  goal,  a  new  direction  was  needed 

•  Strategic  Directions: 

-  New  goal  for  FY  2010 

-  Focus  on  life  cycle  O&S  cost  reductions 

-  Focus  on  institutionalization 

-  Direct  funding  for  long-term  savings  projects 


USD(AT&L)  FY  2010  R-TOC  Goal 


•  USD(AT&L)  Goal:  “Maximize  cost  avoidance  on  total 
defense  systems  FY  2010  O&S  costs  from  an  FY 
2004  baseline,  by  offsetting  30%  of  predicted 
inflation.” 

-  Goal  extends  to  all  defense  systems  on  program-by- 
program  basis 

-  15  Special  Interest  Programs  (SIPs)  designated  lead 
programs  to  “show  the  way”  towards  achieving  the  goal 

-  SIPs  are  monitored  through  semi-annual  reports  and 
quarterly  R-TOC  Forums 

-  Services  will  include  this  goal  in  their  reviews 

•  Ultimately  expand  to  all  defense  systems 

•  $25M/year  R-TOC  PE  created 
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R-TOC 

Special  Interest  Programs  (SIPs) 


Army 

•  Bradley  A3  Upgrades 

•  UH-60M  -  Upgrade 

•  Stryker 

•  UAVS 

•  Guardrail 

Air  Force 

•  Global  Hawk 

•  Engines  (2) 

•  F-16 


Navy 

•  H-1  Upgrades 

•  V-22 

•  F/A-18E/F 

•  H-60 

•  ASE 

•  Common  Ship 

Joint 

•  F-35  (JSF) 


10/29/2008 
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Status  of  R-TOC  SIP  Program  Savings 


700.0% 

600.0% 

500.0% 

400.0% 

300.0% 

200.0% 

100.0% 

0.0% 


■  %  FY2010  Goal 

■  %  of  Projected  LC  Costs  X  10 


■  mr 

1  mil 

ii  hi  limn 

1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16 
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UH-60M  Composite  Tailcone 


Program  Description 

Problem:  The  currently  proposed  metal 
tailcone  for  the  UH-60M’s,  MH-60S’s 
and  MH-60R’s  are  labor  intensive  to 
manufacture  and  require  thousands  of 
parts  and  fasteners. 

Solution:  Incorporate  a  composite 
tailcone  into  the  UH-60M,  MH-60S  and 
MH-60R  fleets. 


Benefits 


Investment/ROI 


•  Cost  savings  of  $60,000.00  per  new 
production  aircraft. 

•  Fewer  parts  and  fasteners 

•  No  corrosion  or  fatigue  maintenance 

•  Weight  Reduction  (50  pounds) 


Investment:  $2.35M 
Life  Cycle  ROI:  33:1 
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Ship’s  Material  Condition  Model 


Overview/Problem 

■  USN  does  not  have  a  consistent  objective 
method  to  determine  material  condition  and  its 
impact  on  mission  /  warfare  area 

■  USN  has  multiple  antiquated  software  tools  and 
systems  to  validate,  screen  and  broker  work 
candidates  depending  on  platform  type  and  coast 


HuM:  |r.-r-  »1  Sctnano  |c»ai*o,rrwrr  3  DaU  Pr©c*«»*<t:  200 2- 

Employment:  07  08  2009  Mo*«i  D«t«  10  18  7005  Oata  UpOato.  10  20  200 


■  USN  has  no  objective  method  to  determine  future 
material  condition  readiness  when  routine 
maintenance  is  not  performed 


Solution 


Investment/ROI 


■  Model  each  ship  using  a  hierarchical  structure  that 
will  show  the  impact  of  each  shipboard  equipment 
on  material  condition  readiness 

■  Provide  a  single  validation,  screening  and 
brokering  tool  for  use  across  all  ship  platforms 


Investment:  $0.5M 
Life  Cycle  ROI:  34:1 


■  Allow  for  a  near  term  predictive  nature  in  modeling 
accounting  for  failure  to  perform  routine 
maintenance 


10 


Intermittent  Fault  Detection  &  Isolation 

System  (IFDIS) 


Overview/Issue 

•  Unable  to  duplicate  discrepancy 
on  No  Fault  Found  (NFF)  LRU’s 

•  Bad  Actor  LRU’s  continued  to  be 
recycled  through  the  repair  cycle 

process 


Solution 

•  Develop  maintenance  tool  to 
augment  traditional  testing 
methods 

•  Will  identify  and  isolate 
intermittent  faults  on  end  items 

•  Repeats  Vigorous  Test  scenario 


Investment/ROI 


Investment:  $2.20M 
Life  Cycle  ROI:  22:1 
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R-TOC  Projects  Cost  Reductions 


FY2006 

FY2007 

FY2008 

FY2009 

FY2010 

Army 

LC  ROI 

34:1 

48:1 

27:1 

64:1 

32:1 

LC  Savings 

$1,730M 

$179M 

$295M 

$714M 

$345M 

DoN 

LC  ROI 

60:1 

35:1 

21:1 

50:1 

61:1 

LC  Savings 

$155M 

$95M 

$359M 

$735M 

$463M 

Air  Force 

LC  ROI 

100:1 

108:1 

33:1 

100:1 

68:1 

LC  Savings 

$2,205M 

$261 M 

$522M 

$557M 

$71 8M 

DoD  Total  ROI 

71:1 

75:1 

28:1 

69:1 

58:1 

DoD  Total  Savings 

$4,090M 

$535M 

$1,176M 

$2,006M 

$1,527M 

DoD  TOTAL  FY06-10 

Life  Cycle  Savings 

$9,334M 

Average  LC  ROI 

80:1 

12 


Contributing  to  R-TOC 


•  Lean  Enterprise  Value 

•  Six  Sigma 

•  Supply  Chain  Management 

•  DoD  Manufacturing  Technology  (ManTech) 

•  Value  Engineering 

-  Law  Requires 

-  FAR  provisions  offer  contractual  incentives 

-  OMB  Directs  Implementation 

-  Strategic  Plan  guides  DoD 

-  Methodology  offers  an  approach  to  partner  with  industry 
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Value  Engineering  is  an 
R-TOC  Best  Practice 


•  VE  provides: 

-  Cost  reduction  (VEPs  and  VECPs) 

-  Product  or  process  improvement 

•  Higher  quality 

•  Reduced  cycle  time 

-  Better  means  and  materials  for  maintenance 

•  Increased  reliability 

•  Greater  safety 

•  Less  environmental  impact 
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Definition 


•  Value  Engineering  -  An  organized  effort  directed  at 
analyzing  the  functions  of  systems ,  equipment, 
facilities,  services,  and  supplies  for  the  purpose  of 
achieving  the  essential  functions  at  the  lowest  life  cycle 
cost  consistent  with  required  performance,  reliability, 
quality,  and  safety.  OMB  Circular  A-1 31 


VE  Goal:  Lower  the  government’s  costs,  improve  value  & 
provide  cost  effective  solutions  to  problems 
in  design,  development,  fielding,  support,  &  disposal 
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VE  Authority 


•  Office  of  Federal  Procurement  Policy  Act  41  USC 
432  -  Each  executive  agency  shall  establish  & 
maintain  cost-effective  VE  procedures  &  processes 

•  Public  Law  Implemented  by  OMB  Circular  A-131 


All  Agencies  Will: 

-  Establish  and  maintain 
a  VE  Program 

-  Develop  annual  plans 

-  Budget  for  VE 


-  Encourage  VECPs 

-  Encourage  VEPs 

-  Identify  and  report  results 

-  Provide  training 


•  OMB  Circular  A-131  implemented  by  the  DoD 
through  VE  Strategic  Plan 
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DoD  VE  Strategic  Plan 


•  Signed  by  USD  (AT&L) 

•  Objectives 

1 .  Improve  the  Value  Proposition  for  Defense 
Systems 

2.  Align  Industry  and  Government  Value 
Propositions  in  Defense  Systems 

3.  Increase  Value  Engineering  Expertise 


SAVINGS  GOAL  =  1.5%  OF  TOA  ANNUALLY 
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DoD  VE  Savings  and  Cost  Avoidance 


83  85  87  89  91  93  95  97  99  01  03  05  07 

Fiscal  Year 


VE  -  An  Industry  Example 


1998  Toyota  Corolla  -  VE  Project 

•  Problems:  Increased  material  costs,  production 
time  issues 


Objective:  Correct  problems  using  VE 


-  Lighter  by  10% 

-  25%  Fewer  engine  parts 

-  Faster  production 

-  Better  fuel  economy 

-  Decreased  emissions 

-  15%  Horsepower  increase 

-  Costs  $1,000  less  to  make  than  in  1997 
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The  Defense  Acauisition  Svstem 


User  Needs 

Technology  Opportunities  &  Resources 

•  The  Materiel  Development  Decision  precedes 
entry  into  any  phase  of  the  acquisition  framework 

•  Entrance  criteria  met  before  entering  phase 

•  Evolutionary  Acquisition  or  Single  Step  to 
Full  Capability 


A 


(Program 

Initiation) 


A 


IOC 


FOC 


Materiel 

Solution 

Analysis 

Technology 

Development 

Engineering  and 
Manufacturing 
Development  & 
Demonstration 

Production  & 
Deployment 

Operations  & 
Support 

^  Materiel 
>  Development 
/  Decision 

/\Post-CDR 

Assessment 

LRIP/IOT&E  A  Decision 

V  Review 

Pre-Systems  Acquisition/ \ 


Sustainment 


Systems  Acquisition 


Decision  Point  /\=  Milestone  Review 


VE  in  Systems  Engineering 


*  VE  methodology  is  an  effective  tool  for  making 
systems  engineering  decisions 

-  Reduce  cost 

-  Increase  productivity 

-  Improve  quality  related  features 

While... meeting  or  exceeding  functional  performance 
capabilities 

•  VE  is  applicable  at  any  point  in  the  life  cycle 

How... making  SE  trades 
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VE  and  R-TOC  in  Systems  Engineering 


*  VE  and  R-TOC  Early  in  the  Life  Cycle  -  Material 
Solution  Analysis 

-  Analysis  of  Alternatives  -  evaluate  functions  vs. 
requirements 

-  Challenge  needs/ensure  requirements  are  valid 

-  SE  trades 

•  Develop  cost  of  alternatives 

-  Consider  life  cycle  cost  implications  -  (R-TOC) 


Savings  For  All  Production  Units 
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VE  and  R-TOC  in  Systems  Engineering 


*  VE  and  R-TOC  During  Technology  Development 

-  Analyze  value  of  requirements/specifications 

•  Can  these  be  tailored? 

-  Cost  as  an  independent  variable 

-  Compare  function,  cost  and  worth  of  technologies 

-  Consider  life  cycle  cost  implications  of  new 
technologies  -  R-TOC 
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VE  and  R-TOC  in  Systems  Engineering 


•  VE  and  R-TOC  During  Engineering  and 

Manufacturing  Development  and  Demonstration 

-  Identify  technical  approaches 

-  Eliminate  unnecessary  design  restrictions 

-  Estimate  cost  of  functions 

-  Identify  alternatives 

-  Evaluate  design  concepts  -  O&S  life  cycle 
concepts  (R-TOC) 

-  Search  for  new  technologies 

-  Simplify  designs 
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VE  and  R-TOC  in  Systems  Engineering 


•  VE  and  R-TOC  During  Production  and  Deployment 

-  Evaluate  and  improve  manufacturing  processes,  methods  and 
materials 

*  VE  and  R-TOC  During  Operations  and  Support 

-  Analyze  advances  in  technologies 

-  Evaluate  modifications 

-  Reduce  repair  costs  -  R-TOC 

-  Analyze  packaging  requirements 

-  Improve  RM&S  -  R-TOC 

-  Analyze/Improve  supply  chain/logistics  footprint  -  R-TOC 

-  Implement  CBM  -  R-TOC 

-  Reduce  manpower- R-TOC 
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SUMMARY 


•  R-TOC  and  VE  provide  savings/cost  avoidances  for  DoD 

•  VE  is  a  tool  for  Systems  Engineering  -  All  Life  Cycle  Phases 

•  R-TOC  provides  a  focus  on  O&S  considerations  -  All  Life 
Cycle  Phases 

•  DoD  VE  documents:  1 )  VE  Contractor’s  Guide,  2)  VECP 
Contracting  Guide,  and  3)  VE  Handbook 

•  VE  revitalization  effort  in-work  - 

-  USD(A&T)  memo  on  compliance  with  OMB  Circular  A-1 31  guidance 

-  Joint  Analysis  Team  (JAT) 

•  OMB  A-1 31  update  needed 

•  R-TOCA/E  websites:  http://rtoc.ida.org  or  http://ve.ida.org 

•  R-TOC  /  VE  Points  of  Contact:  Chet  Bracuto: 

and  Danny  Reed: 
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Air  Force 

Systems  Engineering  Assessment  Model 

(AF  SEAM) 


Randy  Bullard 

AF  Center  for  Systems  Engineering 
Randy.bullard@afit.edu 


1 


Outline 


1. 

2. 


Background 

•  AF  SEAM  Pedigree 


•  AF  SEAM  Goals 

Model  Contents  (What  is  Included) 


•  Process  Areas  (PAs) 

•  Practices  (Specific) 

•  Practices  (Generic) 

•  References  (What) 

•  Other  Information/Elaboration 

•  Typical  Work  Products 

•  Methodology 
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AF  SEAM  Background 


•  In  2006,  AFMC  Engineering  Council  Action  Item  to: 

•  Provide  an  AF-wide  SE  Assessment  Model 

•  Involve  AF  Centers  (product  and  logistics) 

•  Leverage  current  CMMI®-based  models  in  use  at  AF  Centers 

•  Baseline  Process  capability  &  usage 

*  Definition  of  AF  Systems  Engineering  Assessment 
Model: 


•  A  single  AF-wide  tool  which  can  be  used  for  the 
assessment  and  improvement  of  systems  engineering 
processes  in  a  program/project. 
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AF  SEAM  Goals 


•  Goals 


•  Ensure  a  Consistent  Understanding  of  SE 

•  Ensure  Core  SE  Processes  are  in  Place  and 
Being  Practiced 

•  Document  repeatable  SE  “Best  Practices” 
across  AF 

•  Identify  Opportunities  for  Continuous 
Improvement 

•  Clarify  Roles  and  Responsibilities 

•  Improve  Program  Performance  &  Reduce 
Technical  Risk 
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Why  We  Need  SE  Assessment 

AS' 


^wtuteOF 


•  Lack  of  Disciplined  System  Engineering  (SE) 
has  been  a  major  contributor  to  poor  program 
performance 

•  Many  Problems  Have  Surfaced  Repeatedly  with 
AF  Programs 

•  Missed  or  Poorly  Validated  Requirements 

•  Poor  Planning  Fundamentals 

•  Lack  of  Integrated  Risk  Management 

•  Lack  of  Rigorous  Process 

•  Lack  of  Process  Flow  Down 

•  Restoring  SE  Discipline  in  AF  Programs  Is  Key 
to  Improved  Performance  and  Credibility 
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Benefits 


Restoring  Disciplined  SE 

•  Clear  Definition  of  Expectations 

•  Well  Aligned  with  Policy 

Established  Assessment 
Methods  &  Tools 

•  Best  Practices  Baseline 

•  Driving  Improvement 

Moving  towards 

•  Deeper  Understanding  of  SE 
Processes 

•  More  Efficient  Programs 


Why  AF  SEAM 


AF  SEAM  is  a  composite  of  Industry  &  DoD 
SE  best  practices 

•  Maps  to  CMMI  -ACQ  1 .2  &  -DEV  1 .2 

•  Consistent  w/  Industry  and  DoD  guidance 

Advantages  to  using  AF  SEAM 

•  Streamlining  of  CMMI  process  areas  to  AF  programs 

•  AF-centric  w /  end-to-end  life  cycle  coverage 

•  More  focused  document  requires  less  program 
overhead 

•  Does  not  require  SEI  certified  assessors 


Impact  to  AF  programs 

•  Assure  programs  are  achieving  desired  outcomes 

•  Ensure  program  teams  have  adequate  resources 

*  Qualified  People,  Process  Discipline,  Tools/Technology 
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AF  SEAM  Pedigree 


All  AF  product  Centers  selected  and  tailored  some  version  of  the 
Software  Engineering  Institute  (SEI)  Capability  Maturity  Model  Integration 
(CMMI®)  to  baseline  process  institutionalization 

SEI  CMMI®  is  the  Defense  Industry-wide  accepted  method  for  process 
appraisal  and  improvement 

The  SEI  CMMI®  incorporates  principles  and  practices  from  recognized 
industry  and  US  Government  system  engineering  and  related  standards 
such  as: 


AFI  63-1201  Life  Cycle  Systems  Engineering 

Defense  Acquisition  Guidebook,  Chapter  4 


MIL-STD  499 B 
ANSI/EIA  632 
IEEE/EIA  731 
ISO/IEEE  15288 
INCOSE 
IEEE  1220 


System  Engineering 

Processes  for  Engineering  a  System 

Systems  Engineering  Capability  Model 

Systems  Engineering-System  Life  Cycle  Processes 

System  Engineering  Standard 

Application  and  Management  of  the  Systems 
Engineering  Process 
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AF  SEAM  Content 


•  Process  Areas  (PAs) 

•  Goals 

•  Practices 

•  Informative  Material 

•  Description 

•  Typical  Work  Products 

•  Reference  Material 

•  Other  Considerations 


Broadest 

Level  of 
Specificity 

Most 


Principles  &  Objectives 


Tools  & 
Technology 

the  Means 
to  Execute 


Process  & 
Procedures 


the  Glue  that 
Holds  it  Together 
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Why  Focus  on  Process? 


"If  you  can't  describe  what  you 
are  doing  as  a  process,  you 
don't  know  what  you  are 
doing. " 


™  W.  Edwards  Deming 
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AF  SEAM  Elements 


- -  49 

•  10  Process  Areas  (PAs) 

-  Based  in  CMMI  process  area  construct 

-  Conforms  with  AFI  63-1201  &  DAG  Chapter  4 


Process  Areas  (PAs) 

•  Configuration  Mgmt  (CM) 

•  Requirements  (R) 

•  Decision  Analysis  (DA) 

•  Risk  Mgmt  (RM) 

•  Design  (D) 

*  Sustainment  (S) 

•  Manufacturing  (M) 

•  Tech  Mgmt  &  Ctrl  (TMC) 

•  Project  Planning  (PP) 

•  Verification  &Validation  (V) 

•  34  Goals  -  Are  Accomplished  through  the  Specific  Practices 

•  120  Specific  Practices 

•  7  Generic  Practices  (Apply  to  each  Process  Area) 
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AF  SEAM  Practices 


•  Specific  Practices  -  Each  one  applies  to  only 
one  Process  Area 


•  Each  Practice  has  Informative  Material 


•  Description 

•  References 

•  Typical  Work  Products 

•  Other  Considerations 


•  Generic  Practices 

•  Must  be  accomplished  for  each  Process  Area 

•  Ensures  specific  practices  are  executed 

•  Involves  stakeholders 
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AF  SEAM  Practices 


Process  Area 

Goals 

Specific 

Practices 

Generic 

Practices 

Total 

Practices 

Configuration  Mgmt 

3 

8 

7 

15 

Decision  Analysis 

1 

5 

7 

12 

Design 

3 

14 

7 

21 

Manufacturing 

4 

12 

7 

19 

Project  Planning 

3 

15 

7 

22 

Requirements 

4 

13 

7 

20 

Risk  Mgmt 

3 

7 

7 

14 

Sustainment 

4 

15 

7 

22 

Tech  Mgmt  &  Control 

4 

15 

7 

22 

V&V 

5 

16 

7 

23 

Total 

34 

120 

70 

190 
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Sample  Specific  Practice 


•  RMG1P1 


Determine  risk  sources  and  categories 


•  Description:  Establish  categories  of  risks  and  risk  sources  for  the  project 
initially  and  refine  the  risk  structure  over  time  (e.g.,  schedule,  cost,  supplier 
execution,  technology  readiness,  manufacturing  readiness,  product  safety,  and 
issues  outside  control  of  team),  using  Integrated  Product  Teams.  Quantify  the 
risk  probability  and  consequence  in  terms  of  cost  and  schedule. 


•  Typical  Work  Products: 


•  Risk  matrix 


•  Risk  management  plan 

•  Reference  Material:  USAF  Operational  Risk  Management,  AFI  90-901 

•  Other  Considerations:  Consider  using  Acquisition  Center  of  Excellence  Risk 
Management  Workshops  when  needed.  For  manufacturing  risks  consider  the 
capability  of  planned  production  processes  to  meet  anticipated  design 
tolerances.  Include  the  supplier’s  capacity  and  capabilities  in  the  analysis. 
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Generic  Practices 


1.  Establish  and  maintain  the  description  of  a  defined  process 

2.  Establish  and  maintain  plans  for  performing  the  process 

3.  Provide  adequate  resources  for  performing  the  process, 
developing  the  work  products,  and  providing  the  services  of 
the  process 

4.  Assign  responsibility  and  authority  for  performing  the 
process,  developing  the  work  products,  and  providing  the 
services  of  the  process 

5.  Train  the  people  performing  or  supporting  the  processes 
needed 

6.  Monitor  and  control  the  process  against  the  process  plan 
and  take  appropriate  corrective  action 

7.  Review  the  activities,  status,  and  results  of  the  process  with 
higher  level  management  and  resolve  issues 
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Process  Detail  Outline 


A-B 

•  Roles/Responsibilities 

•  Training 

-  Leadership 

-  Self  Assessment 


C-D 

•  Build  Team 

•  Train  team 

•  Logistics  support 

•  Set  schedule 


•  Leadership 
identifies  “area(s)” 
of  self  assessment 


•  Describes  self 
assessment  activity 

•  What  needs  to  be 
accomplished 

•  Capture  data 

•  Presentation  of 
results 


Feedback 


17 


Criteria  for  Methodology 


Facilitate  Self  Assessment 


•  Facilitate  Continuous  Improvement 

•  Provide  insight  into  Program/Project  Processes  &  Capability 

•  Objective  Assessment 

•  Consistent  Near  and  Far  Term  Approach 

•  Provide  Results  that  are  meaningful  for  leadership 

•  Relevant  to  PM/PEO/CC 

•  Simple 

•  Understandable 

•  Graphical 

•  Support  Multi-level  Measurement  &  Reporting 

•  Program/Project,  Squadron,  Group,  Wing,  Center 

•  Resource  Allocation 

•  SE  Process  Improvement 
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Defining  the  Methodology 


Low 


Assessment  Continuum 


•  Hands  Off 

•  Promulgate 
Policy 

•  Directives 

•  Instructions 

•  Checklists 

•  Guidance 

•  Expect 
Compliance 


AF  SEAM 

•  Collaborative 
&  inclusive 

•  Leanest  Possible 
Best  Practices  “Must  Dos” 

•  Clearly  Stated  Expectations 

•  Program  Team  &  Assessor 
Team 

•  Training 

Self  Assessment  of  Program 
with  Validation  Assessment 


Hands  On 

Comprehensive 

Continuous 

Process 

Improvement 

•  Highly  Detailed 
Process  Bibles 

•  Training 

Validation 

Assessment 

•  Deep  Dives 


Assessment  Methods  that  Balance  Time  and  Effectiveness 
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SE  Assessment  Activities 


f  ^ 

Phase  I 
Planning 


V _ ; 

*  Read  Ahead  Package 

*  Logistics  Planning 

*  Training 


V 


J 


f  > 

Phase  II 

Self-Assessment 

v _ w 

•  Self  Assessment 
Training 

•  Project  performs 
self-assessment 

•  Provide  self  - 
assessment  to  review 
team 


_ / 


S - % 

Phase  III 
Independent 
Validation 

•  Team  In-Brief 

•  Project  Brief 

•  Review  Self- 
Assessment 

•  Collaborative 
Interviews 

•  Document 
Reviews 


V _ J 


(  > 

Phase  IV 

Report  Results 

V _ y 

•  Consolidate 
Results 

•  Prepare  final 
report  /  outbrief 

•  Deliver  Final 
Results 


V _ J 
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Assessment  Outputs 


•  Feedback 

•  Lessons  learned  from  assessment  tool 

•  Collaborative  review 

•  Findings 

•  Completed  assessment  tool 

•  Strengths 

•  Improvement  opportunities 

•  Output  metrics 

•  Recommendations 

•  Final  outbrief 
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V  Specific  Practices  Summary 


Not  Applicable 


Percentage 
( of  r/jose 
practices 
scored) 

Specific  Goal  I 
SP  1.1 
SP  1  .2 


**► 


Generic  Practices  Summary 


GP  LEGEND 

PA  LEGEND 

1  1  1 

6-7 

0 

4-5 

<4 

Summary 


Goal  is  to  Continue  to  Improve  Program 
Performance 

•  Too  many  examples  of  program  performance/issues 
being  tracked  back  to  lack  of  SE  discipline 

Long  Term  Goal  -  Revitalize  &  Institutionalize 
Systems  Engineering 

•  Use  SE  “Best  Practices” 

•  Assist  programs  in  achieving  desired  outcomes 

•  Assist  program  teams  in  resource  planning 

•  Qualified  People 

•  Disciplined  Processes 

•  Tools/Technology 
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Back  Up  Slides 


Team  Members 


■w 


Center 

Members 

AAC 

Ian  Talbot 

AEDC 

Neil  Peery,  Maj  Mark  Jenks 

ASC 

Gary  Bailey 

AF  CSE 

Rich  Freeman,  Randy  Bullard 

HQAFMC 

Caroline  Buckey 

ESC 

Bob  Swarz,  Bruce  Allgood 

OC-ALC 

Cal  Underwood,  Bill  Raphael 

OO-ALC 

Jim  Belford,  Mahnaz  Maung 

SMC 

Linda  Taylor 

WR-ALC 

Jim  Jeter,  Ronnie  Rogers 

Spiral  2 


W 


•  Capability  Enhancement 

•  Re-look  process  areas  for  improvements 

•  Further  refine  assessment  methodology 

•  Strengthen  inclusion  of  software 

•  Capture  and  promulgate  best  practices/lessons  learned 

•  Review  scoring 

•  Examine  potential  use  for  SE  health  assessment 

•  Migrate  to  web-based  platform 

•  Resources 

•  Funding 

•  People 

•  Computer  Based  Training 

•  Schedule 

•  Estimated  1 -year  effort 

•  One  member  from  each  Center 

•  Working  Group  meetings  held  approximately  bi-monthly 

•  Lead  POC/Steering  Group 

•  Staff  support 

•  Community  of  Interest 

•  Model  sustainment  (continuous  improvement) 


Scoring  Roll-Up 


Specific  Practice  Assessment  Results 

XXX  Center 


Process  Area 


Scoring  Roll-Up 


Generic  Practice  Assessment  Results 

XXX  Center 


GP1  GP2  GP3  GP4  GP5  GP6  GP7 


Practice  Area 


Implementation  By  Center 


CENTER 


AAC 

AEDC 

ASC 

ESC 


5  AUG  08 -FEEDBACK 


"AAC  began  integrating  AF  SEAM  in  our  established  program  assessment 
process  in  January  2008  and  expects  to  complete  this  integration  in  FY09." 

"We  will  begin  implementing  AF  SEAM  in  October." 

"We  are  creating  a  plan  to  migrate  from  our  current  tool  to  SEAM,  tailored 
with  AFMC  and  ASC  specific  areas  of  interest." 

"We  have  initiated  tailoring  efforts  to  implement  AF  SEAM  by  the  end  of  the 
calendar  year.  We  will  be  working  closely  with  SMC,  our  acquisition  partner 
on  the  tailoring  and  implementation  effort." 


OC-ALC  "Strongly  support,  have  plans  in  place,  ready  to  go!" 

OO-ALC  "We  are  implementing  now." 

SMC  "SMC  plans  to  adopt  AF  SEAM  and  comply  with  related  policies." 

"We'll  begin  implementation  at  Robins  with  pilot  assessments  in  F-15  and 
WR-ALC  Avionics." 
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Development  process  yielded  100%  buy-in 


How  Value  Engineering  (VE)  Enhances 
Diminishing  Manufacturing  Sources  and 
Material  Shortages  (DMSMS)  Solutions 


2008  National  Defense  Industrial  Association 
11th  Annual  Systems  Engineering  Conference 

October  22, 2008 

Dr.  Jay  Mandelbaum 

Institute  for  Defense  Analyses 

4850  Mark  Center  Drive  -  Alexandria,  Virginia  22311-1882 


Outline 


*  Introduction  to  DMSMS 

*  Introduction  to  VE 

*  Relationship  of  the  VE  methodology  to  the 
DMSMS  risk  management  process 

*  Real  VE  examples  for  DMSMS  resolution  options 

*  Conclusions  and  next  steps 


Problems  DMSMS  Addresses 


Technology  improvements:  As  new 
products  are  developed,  the 
technology  used  in  predecessor 
products  becomes  outdated,  making 
it  more  difficult  to  maintain  the  older 
equipment 

Decreasing  demand :  The  parts  needed  to  repair  products 
may  become  more  difficult  and  expensive  to  acquire 
because  fewer  are  produced  as  demand  for  them  decreases 

Non-availability  of  materials:  The  materials  required  to 
manufacture  products  may  no  longer  be  available,  or  they 
may  be  uneconomical  to  procure 


DMSMS  Risk  Management  Process 


Identification  "Sound  the  alarm,"  There  must  be  quick  and  concise  communication 

and  Notification  between  all  relevant  parties  when  a  DMSMS  case  first  occurs. 


Determine  the  scope  of  the  problem,  discerning  which  systems 
Verification  wil1  affected,  and  to  what  extent. 


Options 

Analysis 


Generate  solutions  to  the  problem,  collecting  data  and 
analyzing  case-specific  issues  such  as  cost  and  life 
expectancy.  The  best  solution  may  be  combination  of 
several  traditional  methods. 


Resolution / 
implementation 


Determine  a  best  solution  and  discern 
methods  for  implementing  that  best  solution, 
including  methodologies,  financial  budgets, 
expected  time  frames,  and  specific 
responsibilities  of  the  parties  involved. 


Source:  DMSMS  Guidebook,  p.  3-1. 
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Outline 


*  Introduction  to  DMSMS 

*  Introduction  to  VE 

*  Relationship  of  the  VE  methodology  to  the 
DMSMS  risk  management  process 

*  Real  VE  examples  for  DMSMS  resolution  options 

*  Conclusions  and  next  steps 


What  is  VE? 


•  According  to  Public  Law  104-106  value  engineering  means 
an  analysis  of  the  functions  of  a  program,  project,  system, 
product,  item  of  equipment,  building,  facility,  service,  or 
supply  of  an  executive  agency,  performed  by  qualified 
agency  or  contractor  personnel,  directed  at  improving 
performance,  reliability,  quality,  safety,  and  life  cycle  costs. 

•  Characteristics 

-  Systems  engineering  tool 

-  Contractually  authorized 

-  Employs  a  simple,  flexible  and 
structured  methodology 

-  Promotes  innovation  and  creativity 

-  Incentivizes  contractor  to  help 
government’s  value  proposition 
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An  Actual  VECP  for  the 
Evolved  Sea  Sparrow  Missile  (ESSM) 


Background 

-  The  ESSM  is  an  advanced  a  radar-guided  missile  with  a  hig 
explosive  warhead  used  for  surface-to-air  anti-missile 
defense 

-  A  missile  safe  and  arm  fuze  prevents  an  unintended  launch 
and,  once  launched,  arms  the  warhead  when  the  proper 
stimuli  (e.g.,  speed,  gravitational  force)  are  received 

DMSMS  situation 

-  ESSM  design  called  for  an  obsolete  mechanical  safe  and  arming  fuze 

-  Number  of  suppliers  was  limited  and  costs  were  high 

•  Highly  skilled  artisans  were  needed  for  the  manufacturing  process,  and  much  of  the 
world  fuze  market  had  adapted  to  electronic  fuzes 

The  contractor  proposed  a  VECP  to  replace  the  mechanical  safe  and  arm 
fuze  with  an  electronic  one  adapted  from  the  Sidewinder  missile 

-  Development  and  implementation  costs  were  $1,873,911;  took  approximately  2 
years  to  offset 

-  Total  recurring  cost  savings  equaled  $6,832,000,  which,  when  spread  over  the 
1,600  units  involved,  resulted  in  a  net  savings  per  unit  of  $4,270 

-  Savings  shared  equally  between  the  Navy  and  the  contractor 
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Factors  Leading  to  VE  Solutions 


•  Advances  in  technology 

*  Excessive  cost 

*  Questioning  specifications 

*  Additional  design  effort 

•  Changes  in  user’s  needs 

•  Feedback  from  test/use 

*  Opportunities  for  design 
improvements 

•  Miscellaneous 


Problems  DMSMS  Addresses 


Technology  improvements:  As  new 
products  are  developed,  the 
technology  used  in  predecessor 
products  becomes  outdated,  making 
it  more  difficult  to  maintain  the  older 
equipment 

Decreasing  demand :  The  parts  needed  to  repair  products 
may  become  more  difficult  and  expensive  to  acquire 
because  fewer  are  produced  as  demand  for  them  decreases 

Non-availability  of  materials:  The  materials  required  to 
manufacture  products  may  no  longer  be  available,  or  they 
may  be  uneconomical  to  procure 
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Phases  of  the  VE  Methodology  (Job  Plan) 


*  Orientation  Phase 

*  Information  Phase 

*  Function  Analysis  Phase 

*  Creative  Phase 

*  Evaluation  Phase 

*  Development  Phase 

*  Presentation  Phase 

*  Implementation  Phase 


Often  carried  out  in  a  Workshop  format 


Outline 


*  Introduction  to  DMSMS 

*  Introduction  to  VE 

*  Relationship  of  the  VE  methodology  to  the 

DMSMS  risk  management  process _ 

*  Real  VE  examples  for  DMSMS  resolution  options 

*  Conclusions  and  next  steps 
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Linking  the  two  Methodologies 


Phases  of  the  VE 
Methodology 


Steps  in  the  DMSMS  Risk 
Management  Process 


•  Orientation  ◄ - ► 

•  Information  < - ► 

•  Function  analysis  "*| 

•  Creative 

•  Evaluation 

•  Development 

•  Presentation 

•  Implementation 


Identification  and  notification 
Verification 

Options  analysis 


Resolution/implementation 


There  is  a  strong  synergy 
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Potential  VE  Contributions  to  DMSMS 


•  Finds  innovative  approaches  to  problem 
solving  that  might  not  otherwise  be  considered 
using  the  creative  elements  of  the  VE 
methodology 

•  Incentivizes  DoD  participants  and  their  industry 
partners  to  increase  their  joint  value 
proposition  in  achieving  best  value  solutions  as 
part  of  a  successful  business  relationship 

-  Provides  businesses  with  a  strong  profit-based 
incentive  for  using  its  skilled  engineering 
workforce  to  mitigate  DoD’s  DMSMS  issues 

•  Rewards  contractors  for  making  investments  in 
DMSMS  resolution  options 

•  Allows  the  DoD  to  spread  non-recurring 
engineering  costs  over  time,  making  them  far 
easier  to  fund 
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Noil  Recurring  Engineering  Cost 


Benefits  Realized  Regardless  of  the  DMSMS 

Resolution  Option 


Q 

'n? 


e 


Source:  DMSMS  Guidebook  p.  4-11 
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Outline 


*  Introduction  to  DMSMS 

*  Introduction  to  VE 

*  Relationship  of  the  VE  methodology  to  the 
DMSMS  risk  management  process 

*  Real  VE  examples  for  DMSMS  resolution  options 

*  Conclusions  and  next  steps 
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VE  Contributions  to  an  Existing  Stock 

Approach 


•  Definition 


-  The  current  supplier  utilizes  on-hand  inventories 
or  agrees  to  continue  to  produce  the  item  in 
question 

-  Typically  use  a  life-of-type  or  bridge  purchase 

•  Drawbacks  to  this  approach 

-  Costs  for  material  management  including 
packaging,  storage,  transportation,  shelf  life,  and 
upkeep  of  the  inventory 

-  Difficult  to  estimate  demand 

•  How  VE  can  help 


-  Value  engineering  incentivizes  the  contractor  to 
perform  the  material  management  function  and 
solves  short-term  budget  problems  associated 
with  a  quantity  purchase 
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Standard  Missile  Radome  VE  Example 
_ for  Existing  Stock  Approach 

•  Background 

-  The  Standard  Missile  is  a  surface-to-air  air 
defense  weapon  is  a  fleet  area  air  defense  and 
ship  self  defense  weapon 

-  The  radome  is  a  dome  that  covers  the  radar  on 
the  outside  of  the  missile 

•  DMSMS  situation 

-  There  are  few  radome  suppliers  because  of  the  complexity  involved  in 
finishing  them  to  both  withstand  high  heat  and  acceleration  and  allow  signals 
to  penetrate  without  distortion 

-  Due  to  reduced  program  funding,  the  Navy  halved  its  Standard  Missile 
procurement  rate 

-  If  the  radomes  were  to  be  purchased  on  the  revised  procurement  schedule, 
the  unit  price  would  increase  by  50  percent  due  to  production  slow  down 

-  The  Navy  wanted  to  make  a  quantity  purchase  to  reduce  the  overall  cost, 
however  it  did  not  have  the  resources  in  the  current  fiscal  year 

•  The  contractor  used  a  VECP  to  make  the  quantity  radome  purchase  and 
sell  future  radome  lots  back  to  the  Navy  at  the  lower  price,  thus  leading 
to  significant  savings 

-  Total  savings  was  $1,153,500  shared  equally  by  the  contractor  and  the  Navy 
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VE  Contributions  to  a  Reclamation 

Approach 


Definition 

-  Examines  marginal  or  out-of-service 
equipment  or  supplies  as  a  potential  source 
of  DMSMS  parts 

-  Equipment  that  is  in  a  long  supply,  perhaps 
as  a  result  of  a  planned  product 
improvement  or  modernization  effort  where 
baseline  equipment  could  be  cannibalized 

Drawbacks  to  this  approach 

-  Reclaimed  parts  may  be  unserviceable  or 
damaged 

-  Probably  represents  only  a  short-term 
solution 


How  VE  can  help 

-  Value  engineering  can  play  an  important  role 
in  making  reclamation  feasible 


Artillery  VE  Example 
for  Reclamation  Approach 


Background 

-  The  M795  is  a  155-millimeter  high-explosive  artillery  projectile  with  a  high- 
fragmentation  steel  body 

-  It  provides  increased  effectiveness  against  major  ground-force  threats  at 
greater  ranges  for  anti-personnel  and  anti-materiel  targets 

DMSMS  situation 

-  Because  of  a  world-wide  scrap  steel  shortage,  it  was  difficult  to  maintain  a 
source  for  M795  steel 

A  VE  study  was  initiated  to  develop  a  process  to  reutilize  the  steel  from  a 
large  demilitarization  stockpile  of  surplus  Ml 06  8-inch  projectile  shells 

-  The  steel  could  not  be  reclaimed  directly  since  the 
projectiles  contained  trace  amounts  of  explosives 

-  A  process  was  developed  to  decontaminate  and  mill 
the  surplus  Ml 06  projectiles  to  reclaim  the  steel 

-  M795  production  costs  were  decreased 

-  The  demilitarization  stockpile  was  reduced 

-  Total  cost  avoidance  savings  in  FY  2006  for  the 
197,000  projectiles  processed  amounted  to  $9.2 
million 
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VE  Contributions  to  an  Alternative  Source 

Approach 


Definition 

-  Items  currently  in  production  that  are  form, 
fit,  function,  and  interface  qualified 
replacements  such  as  a  superseding  part 
listed  in  a  specification  or  standard 

-  May  apply  to  aftermarket  or  reverse- 
engineered  sources  (discussed  later) 

Drawbacks  to  this  approach 

-  Same  as  existing  stock 

How  VE  can  help 

-  VE  can  increase  the  efficiency  of  the  new 
suppliers’s  production  process 
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VE  Contributions  to  an  Existing  Substitute 

Approach 


Definition 


-  A  different  part  that  is  currently  being  produced  for  a 
different  application  but  is  (or  can  be  made)  capable 
of  performing  fully  (in  terms  of  form,  fit,  and  function) 
in  place  of  the  DMSMS  item 

Drawbacks  to  this  approach 

-  Non-recurring  engineering  expenses 

-  Market  conditions  may  not  have  a  favorable  outcome 
for  the  new  source 

-  Qualifying  and  testing  the  replacement  item 

-  The  unit  cost  may  be  higher 


How  VE  can  help 

-  Value  engineering  function  analysis  identifies  viable 
options  for  items  to  be  used  as  an  existing  substitute 
and  incentivizes  the  prime  contractor  to  invest  in  them 
--  represents  probably  the  most  prevalent  use  of  VE 
for  DoD  weapon  systems 


Phalanx  VE  Example 
for  Existing  Substitute  Approach 


•  Background 

-  The  Phalanx  Close-ln-Weapon-System  is  a  fast-reaction, 
rapid-fire  20-millimeter  gun  system  that  provides  Navy 
ships  with  a  terminal  defense  against  anti-ship  missiles, 
fixed-wing  aircraft,  small  gunboats,  and  helicopters 

-  A  contract  was  awarded  to  retrofit  Phalanx  with  a  manual 
controller  to  direct  fire  against  targets  of  opportunity 

•  The  contractor  submitted  a  VECP  to  replace  the  standard  military 
controller  with  a  ruggedized  commercial  derivative 

-  On  its  own  initiative,  the  contractor  produced  a  modified  unit 

-  Based  on  the  test  results,  the  contractor  had  confidence  that  the  commercial 
derivative  met  all  of  the  technical  requirements  at  a  lower  cost 

-  The  military  standard  controller  would  cost  $7,600,  while  the  commercial 
derivative  was  only  $2,100 

-  Since  each  gun  required  three  controllers,  net  savings  was  $16,500  per  system 

-  Approximately  $2  million  in  savings  were  shared  by  the  Navy  and  the  contractor 
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VE  Contributions  to  an  Aftermarket 

Approach 


Definition 


-  The  original  equipment  manufacturer  authorizes 
the  assembly  of  an  obsolete  part  and  provides 
necessary  tech  data 


-  A  smaller  company  might  undertake  production 
that  is  no  longer  sufficiently  profitable  for  a 
larger  company  at  a  lower  price;  competition 
also  leads  to  lower  cost 

Drawbacks  to  this  approach 

-  Market  conditions  may  not  have  a  favorable 
outcome  for  the  new  source 

-  Non-recurring  engineering  expenses  will  be 
incurred 


-  The  unit  cost  may  be  higher 
How  VE  can  help 

-  Value  engineering  enables  the  development  of 
viable  aftermarket  sources 


AMRAAM  VE  Example 
for  Aftermarket  Approach 


•  Background 

-  AMRAAM  is  a  fire-and-forget  air-to-air  missile  capable  of 
attacking  beyond-visual-range  targets 

-  The  Inertial  Reference  Unit  (IRU)  accurately  measures 
the  missile  vertical  velocity  and  position  enabling  in¬ 
flight  steering  and  targeting  adjustments 

•  DMSMS  situation 

-  Originally,  there  was  only  one  source  for  this  expensive  item 

-  The  contractor  was  aware  that  others  were  interested  in  furnishing 
this  item,  so  the  contractor  provided  the  requirements  and  helped 
encourage  others  in  the  development  of  the  IRU 

•  The  contract  contained  a  mandatory  VE  program  and  DoD 
recognized  the  value  of  having  a  second  source  for  the  IRU 

-  Approximately  $4  million  in  non-recurring  engineering  costs  were 
required 

-  These  efforts  saved  $2,000  per  unit 

-  The  existence  of  a  second  source  through  the  VECP  probably 
prevented  the  price  of  the  IRU  from  increasing 
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VE  Contributions  to  a  Reverse  Engineering 

Approach 


•  Definition 


•  Drawbacks  to  this  approach 

-  Market  conditions  may  not  have  a  favorable 
outcome  for  the  new  source 

-  Non-recurring  engineering  expenses 

-  The  new  unit  cost  may  be  higher 

-  Intellectual  property  rights 

•  How  VE  can  help 


-  A  producer  obtains  and  maintains  the  design, 
equipment,  and  process  rights  to  manufacture 
a  replacement  item  by  analyzing  the  part’s 
structure,  function,  and  operation 


-  Value  engineering  function  analysis  identifies 
viable  options  for  reverse  engineering  parts 
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Missile  VE  Example 
for  Reverse  Engineering  Approach 


•  Background 

-  A  defense  missile  contractor  had  a  sole-source 
subcontractor  for  a  costly  warhead 

-  The  subcontractor  was  having  problems  meeting  /I 
“insensitive  munitions  capability”  requirements  for 
the  warhead  to  not  explode  in  a  fire  or  if  dropped 

•  With  DoD  cooperation,  a  VECP  was  submitted  to  develop  an 
alternative,  and  less  expensive,  source  for  the  warhead  by 
reverse  engineering 

-  Insensitive  munitions  capability  improved  by  using  a  different 
process  for  making  the  explosive  portion  of  the  warhead 

-  Approximately  $12  million  is  being  invested  to  develop  the 
new  source 

-  Estimated  savings  is  $15,000  per  warhead 

-  Second  source  also  expected  to  control  future  cost  increases 
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VE  Contributions  to  a  Redesign  Approach 


Definition 

-  Either  eliminate  the  need  for  the  part  in  question  or 
replace  it  with  another  -  may  occur  at  many  levels 

•  The  DMSMS  part  itself 

•  The  next  higher  level  configuration  item 

•  An  entire  subsystem 

•  The  end  item  itself 

Drawbacks  to  this  approach 

-  Non-recurring  engineering  expenses  for  building  and  testing 
the  new  production  capability 

-  Qualification  and  certification  to  meet  requirements 
How  VE  can  help 

-  Value  engineering  function  analysis  identifies  viable  minor 
redesign  options  and  it  systematically  identifies  economically 
viable  opportunities  for  a  major  redesign  when  there  is  a  high 
degree  of  interdependence  among  parts 


AMRAAM  VE  Example 
for  a  Major  Redesign  Approach 


•  Early  in  its  production,  the  AMRAAM  missile  used  an 
Analog  Range  Correlator 

-  The  unit  was  scheduled  to  be  replaced  by  a  Digital 
Range  Correlator  as  a  pre-planned  product 
improvement 

-  With  implementation  several  years  in  the  future,  the 
contractor  was  faced  with  producing  the  missile  using 
a  very  difficult  to  build  and  extremely  sensitive  Analog 
Range  Correlator 


•  The  contractor  submitted  a  VECP  to  use  an  Interim  Digital  Range 
Correlator 


-  Implementation  occurred  four  years  in  advance  of  the  pre-planned 
version 


•  Savings 

-  $13,000  per  unit 

-  Government  shared  exceeded  $100  million 

-  Contractor  received  over  $20  million  in  VE  incentives  after  being 
reimbursed  for  approximately  $9  million  in  NRE 
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Outline 


*  Introduction  to  DMSMS 

*  Introduction  to  VE 

*  Relationship  of  the  VE  methodology  to  the 
DMSMS  risk  management  process 

*  Real  VE  examples  for  DMSMS  resolution  options 

*  Conclusions  and  next  steps 
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VE  Enriches  DMSMS  Resolution  Options 


VE  is  an  extremely  powerful  tool  and 
methodology  for 

-  Identifying  a  large  number  of  resolution 
options 

-  Evaluating  their  potential  for  solving 
the  problem 

-  Developing  recommendations 

-  Providing  incentives  for  the 
investments  needed  for  successful 
implementation 


Using  the  VE  methodology  provides  greater  opportunity 
for  developing  and  implementing  innovative  solutions 

to  DMSMS  problems 
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A  VE  /  DMSMS  Partnership  Would  be 

Beneficial 


Nature  of  the  partnership 


-  DMSMS  community  identifies  problems 


-  VE  provides  and  incentivizes  alternative  solutions 

Potential  actions  to  develop  a  partnership 

-  Update  the  DMSMS  Guidebook  with  a  comprehensive 
treatment  of  VE  and  its  application  to  DMSMS 

-  Incorporate  DMSMS  examples  into  the  DAU  VE  distance 
learning  course 

-  Incorporate  DMSMS  into  the  introductory  VE  certification 
training 

-  Establish  a  DMSMS  track  at  the  annual  VE  professional 
society  conference 


-  Maintain  and  strengthen  the  VE  track  at  the  annual  DMSMS 
conference 


-  Augment  the  DAU  DMSMS  distance  learning  courses  to 
include  a  section  on  VE 


-  Include  VE  lessons  in  appropriate  DAU  DMSMS  classroom 
material 


Additional  Actions 


*  Outreach  to  contractors  and 
program  managers 

*  Outreach  to  the  PBL  community 

-  Use  of  Value  Engineering 
Program  Requirement  clause 

*  Potential  DFARS  changes 
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Sources  of  More  Information 


•  Contractual  aspects  of  value  engineering 

-  DAU  CON  236  (online  course) 

-  Value  Engineering  Proposal  Training  Course  -  Ball 
Associates, 

•  VE  methodology 

-  SAVE  International 

-  Certified  facilitators  and  consultants 

•  Publications 

-  Value  Engineering  Handbook 

-  Contracting  Guide  to  Value  Engineering 

-  Value  Engineering  Change  Proposals  in  Supplies  or 
Services  Contracts 

-  Value  Methodology  Pocket  Guide 

•  R-TOC/VE  websites:  or 
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GAO 

Accountability  *  Integrity  *  Reliability 


GAO  Review  of  Best  Practices  for 

Quality  Assurance 


11th  Annual  Systems  Engineering  Conference 

October  21,  2008 


Cheryl  Andrew 
Senior  Defense  Analyst 


Agenda 


•  GAO  Audit  Objectives 

•  Background 

•  Scope 

•  Findings 

•  Conclusions 

•  Recommendations 


GAO 

Accountability  *  Integrity  *  Reliability 


Objectives 


Accountability  *  Integrity  *  Reliability 


•  Identify  the  impact  of  quality  problems  on  selected  DOD  weapon  systems 
and  defense  contractors’  practices  that  contributed  to  the  problems 

•  Identify  practices  used  by  leading  commercial  companies  that  can  be  used 
to  improve  the  quality  of  DOD  weapon  systems 

•  Identify  problems  DOD  faces  in  terms  of  improving  quality 

•  Identify  recent  DOD  initiatives  that  could  improve  quality 


Background 


Accountability  *  Integrity  *  Reliability 


•  A  quality  product  is  one  that  is  delivered 

•  on  time 

•  performs  as  expected 

•  performs  when  need 

•  can  be  obtained  at  an  affordable  cost 

•  MIL-Q-9858A  guided  DOD  quality  efforts  from  the  mid-1960’s  to  the  mid- 
1990’s 

•  DOD  adopted  commercial  standards  (i.e.,  ISO  9001)  in  mid-1990’s 


Accountability  *  Integrity  *  Reliability 


Scope 


Commercial  Manufacturers 

•  Boeing  Commercial 

•  Cummins,  Inc., 

•  Kenworth  Truck  Company 

•  Siemens  Medical  Solutions 

•  Space  Systems/Loral 

Commercial  Customers 

•  American  Airlines 

•  Intelsat 


POD  Weapon  Systems  -Prime* 

•  ASDS  -  Northrop  Grumman 

•  ATIRCM/CMWS  -  BAE 

•  EFV  -  General  Dynamics 

•  F-22A  -  Lockheed  Martin 

•  Global  Hawk  -  Northrop  Grumman 

•  JASSM  -  Lockheed  Martin 

•  LPD-17  -  Northrop  Grumman 

•  MH-60S  -  Sikorsky 

•  PAC-3  -  Lockheed  Martin 

•  V-22  -  Bell/Boeing 

•  WGS  -  Boeing 


*  These  contractors  are  involved  with  over  $1  trillion,  or  about  76  percent  of  the  $1 .5  trillion  DOD  plans  to 
spend  on  weapon  systems  in  its  current  portfolio 


Accountability  *  Integrity  *  Reliability 


Objective  1 :  DOD  Quality  Problems  and  Prime 
Contractor  Practices  that  Contributed  to  Problems 


•  For  the  1 1  programs  we  reviewed,  quality  problems  resulted  in 

•  Over  $1 .5  billion  in  cost  overruns 

•  Up  to  5  years  of  schedule  delays 

•  Reduced  weapon  system  availability 

•  Military  personnel  deaths 

•  Prime  contractor  practices  that  contributed  to  problems: 

•  Poor  systems  engineering  practices  related  to  requirements  analysis, 
design,  and  testing 

•  Manufacturing  processes  not  in  control 

•  Supplier  quality  problems 


Accountability  *  Integrity  *  Reliability 


Objective  1 :  Expeditionary  Fighting  Vehicle 
Example  of  Systems  Engineering  Problem 


Contractor  was  only  able  to  demonstrate  7.7 
hours  between  operational  mission  failures 
during  pre-production  testing,  well  short  of 
the  17  hour  goal 


Primary  problem  was  part  and  subsystem 
interferences 


Root  causes  •  4-year  extension  to  SDD 

•  subassembly  teams  claiming  the  same  . *  *750  mj||ion  cost  growth 
space 

•inconsistent  computer  model  checks 

•  lack  of  design  engineer  experience 
•tight  engineering  model  release  schedules 


Objective  1:  LPD-17 

Example  of  Manufacturing  Problems 
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Over  5,000  quality  problems  were  found 

•  Faulty  hydraulics  piping  welds  due  to 
inexperienced  workers  and  improper 
documentation 

•  Some  rework  was  required 

•  All  welds  had  to  be  re-inspected 

•  Could  have  resulted  in  injuries 

•  Peeling  non-skid  coating  due  to  unclean 
surfaces  and  high  humidity 

•  Rework  was  required 

•  Long-term  solution  has  not  been 
identified 


•  3-year  delay 

•  $846  million  cost  growth 


Accountability  *  Integrity  *  Reliability 


Objective  1 :  Patriot  Advanced  Capability-3 
Example  of  Supplier  Quality  Problem 


Program  has  experienced  a  number  of  problems 
with  the  seeker  portion  of  the  missile 

A  sub-tier  supplier  accepted  non-conforming 
hardware  without  authority 

•  seeker  contractor  identified  quality  problem 

•  resulted  in  rework 

•  re-inspection  of  components 


Same  supplier  also  had  poor  workmanship  and 
inadequate  manufacturing  controls 

•  Operated  in  a  development  rather  than  a 
production  environment 


•  6-month  schedule  slip 

•  Delivery  delay  of  100 
missiles 


•  Facility  was  temporarily  shut-down  to  address 
management  and  production  problems 


Accountability  *  Integrity  *  Reliability 


Objective  2  -  Commercial  Best  Practices  - 
Systems  Engineering 


Ensure  that  a  product’s  requirements  are  achievable  with  available  resources 
and  technologies 

•  Siemens  Medical  Solutions 

•  Clear,  precise,  measurable,  comprehensive  requirements 

•  Quality  and  reliability  requirements  prior  to  commitment 

•  Boeing  Commercial  Airplanes 

•  “Mistake-proof  designs 

•  Rating  tool  on  critical  designs 

•  Space  Systems/Loral 

•  Reliability  assessments 

•  Highly  accelerated  life  testing 


Accountability  *  Integrity  *  Reliability 


Commercial  Best  Practices  -  Manufacturing 


Ensure  that  a  product’s  requirements  can  be  produced  consistently  with  high 
quality  and  low  variability 

•  Cummins,  Inc. 

•  Capability  growth  plan  for  manufacturing  processes 

•  Prototypes  to  validate  design  and  production  processes 

•  Kenworth  Truck  Company 

•  Electronic  system  for  process  documents 

•  Pictures  and  engineering  specifications 

•  Training  audits 
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Commercial  Best  Practices  -  Supplier  Quality 


Ensure  that  suppliers  have  the  ability  to  deliver  high-quality  parts 

•  Kenworth  Truck  Company 

•  Hold  first-tier  suppliers  accountable  for  quality  problems  attributed  to 
lower-tier  suppliers 

•  Boeing  Commercial  Airplanes 

•  99%  part  conformance  expectations  for  suppliers 

•  Retain  higher-performing  suppliers 

•  Siemens  Medical  Solutions 

•  98%  part  conformance  expectations  for  suppliers 

•  Levy  financial  penalties  against  non-conforming  suppliers 
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Objective  3  -  Problems  DOD  Faces  When 
Trying  to  Improve  Quality 


•  Environment 

•  DOD  awards  cost  reimbursement  contracts  assumes  most  of  the 
financial  risks 

•  Reliability  is  not  emphasized  at  development  start 

•  Requirements  are  set  without  adequate  systems  engineering 
knowledge 

•  Oversight 

•  Risk-based  approach  used  to  oversee  contractors 

•  DCMA  and  service  oversight  varies  by  program 

•  Information  is  not  aggregated  in  a  manner  that  would  allow  DOD  to 
determine  overall  weapon  system  quality,  prime  contractor 
performance,  or  systemic  problems 
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Objective  4  -  DOD  Initiatives  that  Could 
Improve  Quality 

•  Concept  Decision  Reviews 

•  Time-Defined  Acquisition 

•  Configuration  Steering  Boards 

•  Key  Performance  Parameters/Key  System  Attributes 

•  Award  and  Incentive  Fees 

•  Establishing  Reliability  Goal  and  Demonstrating  Reliability  Prior  to 
Production 

•  New  Reliability,  Availability,  and  Maintainability  Policy  (7/08) 


Conclusions 
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•  Despite  adopting  commercial  quality  standards  and  implementing  new 
requirements  and  systems  engineering  policies,  DOD  still  has  difficulty 
acquiring  high-quality  weapon  systems  in  a  cost-efficient  and  timely 
manner 

•  Poor  systems  engineering,  manufacturing  control,  and  supplier  quality  are 
the  underlying  problems 

•  Improvements  in  analyzing  requirements  and  successful  implementation  of 
several  new  initiatives  could  improve  outcomes 


It  is  going  to  take  a  joint  effort  between  DOD  and  prime 

contractors  to  improve  weapon  system  quality 


Recommendations 
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•  As  part  of  the  concept  decision  review  initiative,  require  systems 

engineering  analysis  be  completed  by  the  prime  contractor  prior  to  entering 
into  a  development  contract 


•  Establish  measures  to  gauge  the  success  of  the  concept  decision  review, 
time-defined  acquisition,  and  configuration  steering  board  initiatives 

•  Identify  and  collect  data  that  provides  metrics  about  the  effectives  of  prime 
contractors’  quality  management  system  by  weapon  system  and  business 
area  over  time 

•  Develop  evaluation  criteria  that  would  all  DOD  to  score  the  performance  of 
contractors’  quality  management  systems  based  on  actual  performance 


The  Role  of  T&E  in  the  Requirements 
Process  for  System  of  Systems 


http://www.ctc.com/learnaboutctc/SoSCE.cfm 


Walter  C.  Reel 
Test  Engineer 
NSWCDD  -  W33 
walter.reel@navy.mil 
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The  Problem 


Charlie  Chaplin  in  “Modern  Times” 


How  do  we  define  testable  requirements  for  System  of  Systems  (SoS) 
when  no  one  understands  exactly  how  the  complex  system  will  operate 
and  integrate  once  it  comes  on-line  and  the  human  in  the  loop  is  added 

to  the  equation? 


The  Problem 


http://thinkingproblemmanageme 
nt.blogspot.com/2008/03/differenc 
es-between-problems-and.html 


Most  problems  with  SoS  designs,  (as  with  most  designs),  lead  us  back 
to  the  requirements  phase. 


The  synthesis  of  these  very  large  systems  often  results  in  different  problems  than 
those  presented  by  the  design  of  a  single,  but  complex,  system. 


Historical  Approach 


In  the  past  the  contribution  of  Test  and  Evaluation  professionals  has  not  come  until 
after  the  system  Detailed  Design  phase. 

It  is  our  recommendation  that  this  be  changed  and  T&E  personnel  be  involved  from 
the  beginning  of  the  Requirements  and  Architecture  phase. 


The  Problem 
Continued 


What  do  we  want  the  SoS  to  be  able  to  do? 


This  is  often  a  very  complex  question  that  can  have  multiple  and  vague  answers 
that  have  little  meaning  when  it  comes  to  defining  measurable  metrics  for  later 
testing  of  our  system. 


We  usually  end  up  with  requirements  that  are  too  detailed  and  “Pie  in  the  Sky” 
requirements  that  are  too  vague  to  implement. 


“The  system  CPU  will  operate  at  500  megahertz.” 


“The  system  will  create  synergy  among  multiple  sensor  systems  and  enable 
data  fusion  at  all  levels.” 


End  User  Example 


Primitive  Need  Statement  for  C2  Cell  of  a  C4ISR  System 


“I  need  to  be  able  to  visualize  what  my  Intel  guys  are  collecting  and  analyzing 
so  that  my  understanding  of  the  battle  area  is  current  and  my  decisions  are  based 
on  accurate,  comprehensive,  up  to  date  information.  I  also  need  to  have  an 
understanding  of  what  is  changing  now,  how  long  things  can  be  expected  to  remain 
the  same  (Dwell  Time)  and  the  status  of  the  enemy’s  assets  as  well  as  my  own, 
and  I  need  this  to  be  a  simple  process.” 
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End  User  Example 


http://www.optemax.com/html/productmilindoor.html 

Seems  to  be  a  reasonable  request,  right? 

Well  let’s  look  at  this  primitive  need  statement  and  try  to  do  a  quick  and  dirty 
breakdown  of  what  functionality  the  Commander  is  looking  for  us  to  incorporate 
into  our  C4ISR  system  to  provide  these  required  Command  and  Control  capabilities. 

• Situational  Awareness : 

Visualization  of  the  Battlespace 
>Near/Real  Time  Information 
>Sensor  Availability 
>Data  Fusion 
>Predictive  Intelligence 
>Blue  Force  Intel 

>Order  of  Battle  (OOB)  7 

>  Advanced  HSI 


End  User  Example 


http://www.cctcorp.com/ 

Now  let’s  break  down  each  of  these  major  functionalities 
into  some  of  their  supporting  functions. 

•Visualization: 

>Maps 

>  Overlays 
>Terrain  information 
>Weather  Information 
>Symbology 

>  Movement  Representation  (Vector) 
>Detail  Drill  Down 

information  Filtering  and  Manipulation 
>DATA  Handling 


End  User  Example 


http://blog.businessquests.com/marketing_marketing_xO/index.html 


•Near/Real  Time  Information: 

> Direct  Sensor  feed 
>Single  Step  Data  Sharing 
> Prioritization  of  Information 
> Latency  of  Information 
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End  User  Example 


•Sensor  Availability: 

^Multiple  Sensors 

>On  Station  Time 

>Full  Spectrum  of  Sensor  Types 

>Local  Sensor  Tasking 

>Live  Sensor  Data  Feeds 

> Information  on  Data  Accuracy/Latency 


End  User  Example 


•Data  Fusion: 

>  Autonomous  Data  Fusion 
>Selectable  Data  Fusion 
> Fusion  C2  Products 
>Fusion  C2  Symbology 
> Information  Reliability  Ratings 
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End  User  Example 


•Predictive  Intelligence: 

>Dwell  Time 

>Probable  Destination  for  Moving  Units 
>Probable  Unit  Strength 
>Probable  Unit  Type 
>Probable  Unit  Action 
>Predicted  Unit  Weaknesses 
information  Reliability  Ratings 


End  User  Example 


•Blue  Force  Intel: 

>Unit  Location 
>Unit  Movement 
>Unit  SITREP 
> Latency  of  Information 
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End  User  Example 
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•OOB: 

>Known  Enemy  Unit  Locations 
>Known  Enemy  Unit  Equipment 
>Known  Enemy  Unit  Strength 
>Known  Enemy  Unit  Weaknesses 
>Known  Enemy  Unit  Range  and  Speed 
> Latency  and  reliability  of  Information 
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End  User  Example 


Human 

http://dfs.iis.u-tokyo.ac.jp/~niitsuma/ 


•Advanced  HSI: 

>Operators  Control  Visual  Clutter 

>Simple  HSI  Actions  for  Data  Manipulation,  Retrieval,  and  Storage 

We  can  readily  see  that  many  systems  will  be  involved  in  providing  these 
capabilities  to  the  Command  and  Control  Cell  to  meet  the  user’s  needs. 
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End  User  Example 


Generate  an  Effective  Need  Statement 


“A  Command  and  Control  System  is  required  that  integrates 
Near/Real  Time  Information  from  Enemy  OOB,  all  deployed 
Sensors,  ISR  Data  Fusion,  Intelligence  Analysis,  Predictive 
Intelligence,  Blue  Force  Intel,  enemy  unit  location,  and  all 
unit  movement  data.  The  system  will  allow  Visualization  of 
this  information  within  the  defined  Battlespace  and  allow  the 
operator  to  manipulate  and  request  information  updates  and 
details  utilizing  simple  HSI  functionality.  The  system  will  be 
able  to  filter,  store,  and  transfer  information  for  detailed 
scrutiny  to  limit  visual  clutter.” 
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What’s  Next 


The  Requirements  Definition  process  now  becomes  our  primary  mission 

This  process  should  be  conducted  utilizing  the  Integrated  Product  Team  approach 
and  should  include  all  Stakeholders. 

If  the  requirements  are  too  vague  System  Design  will  suffer  as  will  construction, 
test  and  evaluation. 


How  (he  system  was  designed  As  the  programmer  wrote  it 


What  the  user  really  wanted  How  it  actually  works 


If  the  requirements  are  too  specific  the  ability  of  the  contractor  to  build  a  better 
mousetrap  will  be  hindered  and  system  functionality  may  suffer 
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Requirements  Generation 


What  approach  will  ensure  that  the  requirements  are  written  so  that  they  properly 
support  the  user’s  needs  and  also  provide  design  and  testing  adequate  information 
to  do  their  job? 

Requirements  should  be  written  and  then  evaluated  and  then  re-written  and  then 
re-evaluated  and  then  re-written  and  re-evaluated,  etc...,  until  a  consensus  is 
reached  by  ALL  Stakeholders. 


We  are  reminded  of  the  old  carpenter’s  adage,  “measure  twice  and  cut  once.” 
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Collaborating  System  Interactions 


In  a  SoS  world,  requirements  may  already  be  established  for  the  sytems  that 
you  will  integrate  with. 


This  can  often  require  great  negotiating  skills,  if  you  are  the  new  kid  on  the 
block  it  is  very  likely  that  you  will  have  to  make  most  of  the  concessions  if 
there  are  issues  with  interfacing  with  fielded  systems. 
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Requirements  Generation 


http://www.automation.si 

emens.com/mc/mc- 

sol/en/9279dd53-befd- 


4935-827a- 

11459b1a2863/index. 


Here  we  see  an  inherent  problem  with  SoS  design  and  Acquisition: 

If  we  have  to  incorporate  or  modify  existing  systems  in  order  to  achieve  the 
desired  functionality  to  fill  the  user’s  need  our  process  will  become  more  time 
and  coordination  intensive. 


This  fact  has  driven  the  requirements  for  Net-Centric  Design  and  Service 
Oriented  Architecture  that  could  ease  the  integration  of  multiple  Inter/Intra¬ 
network  systems  into  a  SoS  framework 
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Requirements  Generation 


Let’s  get  back  to  our  discussion  and  look  at  Information  Exchange  Requirements 
(lERs).  In  a  SoS  acquisition  lERs  become  much  more  important  than  in  most 
system  designs. 

lERs  tell  us  who  exchanges  what  information  with  whom,  why  the  information  is 
necessary,  how  the  information  is  used,  and  defines  the  metrics  for  the  IER. 

In  SoS  there  may  be  hundreds  of  lERs  and  to  imagine  having  the  resources  to  test 
and  evaluate  all  of  them  is  unrealistic.  Just  as  we  cannot  possibly  test  all  possible 
combinations  of  inputs  and  pre-conditions  in  a  complex  software  program  we  will 
not  be  able  to  test  all  lERs  in  a  complex  SoS.  Therefore,  as  with  software,  we  must 
evaluate  the  SoS’s  states  and  behaviors  against  a  specification. 
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Requirements  Generation 

DoD  Architecture  Framework 
Version  1.5 


Ail  Mew 


Core  Architecture  Data  Model 


A  key  tool  for  the  definition  of  how  a  System  of  Systems  will  interface,  who  it  will 
interface  with  and  what  data  will  be  exchanged  and  the  rules  for  exchanging 
information  among  systems  is  the  Department  of  Defense  Architectural 
Framework  (DODAF).  The  major  product  areas  are  defined  in  the  figure  above. 


Input  in  the  generation  of  these  documents  by  Test  and  Evaluation  personnel 
would  help  to  insure  that  they  can  be  realistically  tested  and  also  provide  T&E 
experience  to  the  sponsor  during  the  early  generation  of  system  capability  definition. 


The  Role  of  Test  and  Evaluation 


T&E  personnel  must  be  involved  from  the  beginning  of  the  Requirements 
Definition  Phase 


Test  personnel  will  be  responsible  for  designing  the  developmental  and 
operational  testing  of  the  system  and  need  to  have  input  into  whether  or 
not  the  requirements  being  generated  can  be  tested. 
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The  Role  of  Test  and  Evaluation 


Requirements  come  from  many  sources; 

Sponsors  put  them  in  the  Acquisition  Capabilities  documents  when  defining  for  the 
acquisition  Program  Manager  their  Key  Performance  Parameters  (KPPs)  and  other 
attributes. 

Requirements  may  be  determined  by  the  Joint  Chiefs  of  Staff  (JCS)  as  overarching 
requirements  for  all  programs. 


Requirements  are  derived  from  discussion  with  stakeholders  and  users  and  they 
evolve  from  the  process  of  defining  the  required  system  performance. 
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The  Role  of  Test  and  Evaluation 


Requirements  are  transformed; 


They  become  Critical  Technical  Parameters  (CTPs)  and  Software  Design  or  System 
Specifications  (SDS/SS). 


Systems  are  built  and  tested  based  on  specifications,  whether  they  be  software  or 
system. 


Specifications  define  how  a  system  must  function  and  support  system  requirements. 

If  requirements  are  so  vague  that  they  cannot  be  supported  by  specifications  how  can 
the  requirement  be  met  or  tested? 


The  Role  of  Test  and  Evaluation 


Testing  SoS  requires  that  the  system  be  evaluated  for  acceptability  by  the 
end  users,  the  target  audience,  the  purchasers,  and  other  stakeholders. 

It  is  resource  intensive  to  test  all  individual  lERs,  we  must  rely  on  testing  the 
outcome  of  their  contribution  to  the  system,  as  defined  in  our  specifications, 
to  evaluate  the  systems  overall  acceptability. 

We  must  know  what  will  not  be  tested  as  part  of  our  evaluation  in  order  to  examine 
whether  this  imposes  risk  to  the  performance,  or  our  stakeholders  acceptance  of  the 
system. 


The  Role  of  Test  and  Evaluation 


http://www.toronto.drdc- 
rddc.gc.ca/facilities/spe 
cialized  e.html 


Other  areas  for  requirements  that  need  to  be  addressed  are  the  Humans  in  the  Loop 
(HITL)  and  Human  System  Interfaces  (HSI). 

These  provide  a  whole  set  of  other  requirements  and  can  be  resource  intensive  to 
evaluate.  If  our  SoS  require  HITL  at  several  different  systems’  locations  the 
complexity  of  the  testing  is  increased  greatly. 

The  performance  of  most  systems,  however;  is  tied  to  the  performance  of  HITL  and 
to  ignore  these  requirements  would  add  risk  to  the  systems  acceptability. 
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The  Role  of  Test  and  Evaluation 


http://www.bbc.co.uk/wear/content/image_gal 

leries/sunderland_airshow_yourpics2_gallery 

.shtml?50 


Another  dimension  of  HITL  that  is  not  normally  designed  into  testing  is  the  ability  of 
personnel  to  utilize  the  system  in  ways  that  had  not  been  imagined  when  the 
system  was  designed. 

This  can  be  a  positive  or  a  negative  factor  but  is  a  valuable  input  into  the  systems’ 
readiness. 

It  is  important  that  testing  is  not  always  strictly  scripted  to  give  the  operators 
latitude  when  conducting  their  portion  of  the  evaluation.  Doing  this  will  often  lead  to 
discoveries  testers  had  not  anticipated! 


Summary 


“Interdependence” 


http://www.modernartwork.net/gallery_Modern_Art_Work_drawings.htm 

The  interdependence  of  Requirements  Definition,  Test,  and  Evaluation  takes  place 
throughout  the  program  cycle. 

To  exclude  T&E  from  the  beginning  stages  of  requirements  development  is  to 
preclude  more  opportunity  for  synergy. 

To  operate  in  an  efficient  manner  all  components  of  a  system  must  work  together; 
this  is  a  basic  definition  of  a  system.  Why  would  we  as  the  systems  engineers  want 
to  deny  this  basic  truth  and  eliminate  the  T&E  capability  of  our  acquisition  system? 
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Joint  Test  and  Evaluation 
Methodology  (JTEM) 


Systems  Engineering  for  Testing 
in  a  Joint  Mission  Environment  (JME) 

National  Defense  Industrial  Association 
11th  Annual  Systems  Engineering  Conference 

Test  &  Evaluation  in  Systems  Engineering  Track 

October  22,  2008 


Earl  Reyes 

Senior  Systems  Engineer 
757.638.6014 
earl.reves@ite.osd.mil 


Overview 


•  JTEM  Problem  Statement 

•  Capability  Test  Methodology  (CTM) 

•  CTM  Systems  Engineering  Thread 

•  Summary 


item@ite.osd.mil 
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•  JTEM  Problem  Statement 

•  Capability  Test  Methodology  (CTM) 

•  CTM  Systems  Engineering  Thread 

•  Summary 


item@ite.osd.mil 
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JTEM  Problem  Statement 


Processes  and  methods  for  designing  and  executing 
tests  of  systems  of  systems  in  the  joint  mission 
environment  are  not  well  defined  or  understood.  Nor  is 
there  a  clear  understanding  of  how  to  assess  system 
performance  as  it  pertains  to  capabilities  supporting  joint 
missions. 


Overall  Goal:  Recommended  Best  Practices  for  a  consistent 
apgmach  to  describing,  building,  and  using  an  appropriate 
representation  of  a  particular  Joint  Mission  Environment 

across  the  acquisition  lifecycle. 
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Overview 


•  JTEM  Problem  Statement 

•  Capability  Test  Methodology  (CTM) 

•  CTM  Systems  Engineering  Thread 

•  Summary 
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4.  Manage  Test 
Execution 


Joint 

Capability 

Evaluation 

(JCE) 


Event 

Management 

Plan 


JTEM  Capability  Test  Methodology 

(CTM)  v2.0 


6  Steps 
14 JTEM 
Processes 


0.  Develop 
T&E  Strategy 


T&E 

T&E 

Strategy 

Master 

(TES) 

Plan 

(TEMP) 

Develop  Capability/SoS 
Description 

Develop  Joint  Operational 
Context  for  Test  (JOC-T) 
Develop  Evaluation  Strategy 
Develop/Refine  Capability 
Crosswalk 


1. 

Characterize  Test 

Program 

Statement 

Introduction 

of 

Document 

Capabilities 

(PID) 

(SOC) 

•  Develop  Test  Concept 

•  Refine  Evaluation  Strategy 

•  Technical  Assessment 

Capability 
Subset 
Focus 


Test 

Concept  *. 

A 

Joint  Operational 

Context  for  Test 

. 


Test 

Data 

( 


Capability 

Set 

Focus 


5.  Evaluate  Capability 


LVC  -  Live,  Virtual,  Constructive 
LVC-DE  -  Live,  Virtual,  Constructive 
Distributed  Environment 
SoS  -  System  of  Systems 


Analyze  Data 

Evaluate  SoS  Performance  & 
Joint  Mission  Effectiveness 


2.  Plan  Test 


Test 


Plan 


Develop  Test  Design 
Perform  LVC  Distributed 
Environment  Analysis 
Develop  Test  Plan 


Integrated 
*  Vignettes 

LVC 

Distributed 

Environment 

Design 

Joint  Mission 
Environment 

Test  Control  & 
>.  Monitoring 


3.  Implement 
LVC-DE 


System 

Design 

Document 

(SDD) 


Design  LVC  Distributed 
Environment  Configuration 
Integrate  LVC  Distributed 
Environment 


Event 

Focus 
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Overview 


•  JTEM  Problem  Statement 

•  Capability  Test  Methodology  (CTM) 

•  CTM  Systems  Engineering  Thread 

•  Summary 
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What  is  the  Joint  Mission 
Environment? 


non/  ,0  Analytic  Agenda  JOpsC  Family 

OSD/ Jo  J 7  DPS,  MSFD,FYAB  -- ___  JOC,  JFC,  JIC 


JS  J8/J7  JCD,  ICD,  CDD,  JCIDSDoDAF*Vjews  (e.g.,  OVs,  SVsJ 

“Capability  Lead”  aoa 


USJFCOM  J8/J7  JCAs 

UJTL/Service  Task  Lists 

AT&L/ 

Service/Joint  Acq  Programs 


Joint  Operational 
Context  for  Test 
(JOC-T) 


Mission 
Blue 
Threat 

Environment 
Interactions 
JOC-T  DoDAF  Views 


JTEM 


Methods  and 
Processes 


JMETC 


Infrastructure 


Components  usn/mc 


L/Ve,  Virtual^  Constructive 

Distributed  Environment 

(LVC-DE) 

JME  Foundation  Model  (JFM) 

JME  DoDAF  Views 

Models, 

Middleware,  Infrastructure 


JMIE 

instantiated 
for  a 

capability  test 
execution 
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Sea  Environmer 
Air  Componer 
Surface  Compoi 
Subsurface  Comp 
CONOPS 
Link  Models 
C2/ISR 
Space 


AF-ICE  CORE 

USAF  Air  Component 
USAF  C2/ISR 
USAF  Weapon 
USAF  Link  Mo 
USAF  CONOi 
IADS  Tools 
Space 
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System  Engineering  Process 


m 


CTM  3.4 
Integra  te 
LVC  DE 


LVC-  Configuration 
DE  Analysis 


0.2 
JOC-T 


Assessment 


JFM 


Logical 

Physical 


*  Abstract 

♦  Conceptual 

•  Template 

*  Use  Case 

*  Implementation 
Independent 

•  Actual  systems 

•  All  layers 


JFM  identifies  generic 
capabilities,  environments, 
behaviors,  and  tasks 

Logical  Design  applies  the  JFM 
to  a  use  case  (e.g.,  JCAS, 
JFIRES,  etc.)  independent  of 
implementation 

Logical  Design  is  transformed 
into  a  physical  design 

Physical  interfaces  transformed 
to  executable  software 

Physical  Design  solutions  are 
integrated  into  a  JME  (moves  to 
LVC-DE  repository  for  reuse) 
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JTEM  Capability  Test  Methodology 

(CTM)  v2.0 


6  Steps 
14 JTEM 
Processes 


0.  Develop 
T&E  Strategy 


T&E 

T&E 

Strategy 

Master 

(TES) 

Plan 

(TEMP) 

Develop  Capability/SoS 
Description 

Develop  Joint  Operational 
Context  for  Test  (JOC-T) 
Develop  Evaluation  Strategy 
Develop/Refine  Capability 
Crosswalk 


1. 

Characterize  Test 

Program 

Statement 

Introduction 

of 

Document 

Capabilities 

(PID) 

(SOC) 

•  Develop  Test  Concept 

•  Refine  Evaluation  Strategy 

•  Technical  Assessment 

Capability 
Subset 
Focus 


2.  Plan  Test 


Develop  Test  Design 
Perform  LVC  Distributed 
Environment  Analysis 
Develop  Test  Plan 


Test 

Concept  *. 

A 

oint  Operational 
Context  for  Test 

•  •••••••  <£►  j 


Capability 
Set 
Focus 


Test 
Data 

< 

JL_ 

5.  Evaluate  Capability 


Integrated 
*  Vignettes 

LVC 

Distributed 
Environment 
Design 


LVC  -  Live,  Virtual,  Constructive 
LVC-DE  -  Live,  Virtual,  Constructive 
Distributed  Environment 
SoS  -  System  of  Systems 
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Monitoring 

_ • _ 

4.  Manage  Test 
Execution 


3.  Implement 
LVC-DE 

i 

System 

Design 

Document 

(SDD) 

■  •  Design  LVC  Distributed 
Environment  Configuration 
•  Integrate  LVC  Distributed 
t  Environment 

Event 

Focus 

Analyze  Data 

Evaluate  SoS  Performance  & 
Joint  Mission  Effectiveness 


10 


JME  Foundation  Model  (JFM) 


•  Conceptual  model  to  enable  implementation- 
independent  reasoning  in  an  idealized  framework 

•  Provides  abstract  interface  descriptions  and  the 
logical  and  quantitative  relationships  between 
those  interfaces 

•  The  goal  of  the  JFM  is  to  provide  a  frame  of 
reference  for  LVC-DE  configuration  design 

•  The  JFM  Description  is  an  evolutionary  document 
that  will  be  modified  over  time  to  promote  the 
robustness  of  the  JME 

The  JFM  is  a  design  template  to  guide  the 
development  and  reuse  of  LVC-DE  systems. 
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CTM  0.2:  Develop  JOC-T 
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CTM  0.2.1:  Analyze  Mission 

Objectives 


Recommended  Input  Info/Sources 


Products 


0.2  Develop  JOC-T 
□ 


Joint  Operational  Context  for  Test 
(JOC-T) 


>  « 


2.1  Analyze  Mission 


Concent  of  Operations  Summary 


Analytical  Baseline  (DPS,  MSFD,  FYAB) 
OV-1  Hiah-Level  Operational  Concent 
Objectives 


I 


□  |  Operational  Overview 


u 


0.2.5  Compose  Joint  Operational  Context 


< 
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CTM  0.2.2:  Analyze  Blue 


(Example 


Recommended  Input  Info/Sources 


Products 


Blue  High-Level  Graohic  (BOV-1) 


Blue  Operational  Node  Connectivity  Model  (BOV-5) 


Blue  Operational  Node  Connectivity  Description 
(BOV-2) 

Blue  Notional  Relationship  Chart/Task  Organization 


(BOV-4) 


Blue  SoS  Interface  Description  (BSV-1) 

Blue  SoS  Operational  Activity  to  Svstem/SoS 
Function  T raceabilitv  Matrix  Description  (BSV-5) 

Blue  SoS  Functionality  Description  (BSV-4) 


Blue  SUT  Information  Exchange  Matrix  (BSV-6) 


0.2  Develop  JOC-T 

□(  Joint  Operational  Context  for  Test  (JOC-T) 


Lc 


Joint  Tasks 

Forces  and  Related  Conditions 

Tasks  Steps/Mission  Threads  (JTA,  other  sources) 

Capability  Discussion 

Svstem/SoS  Capabilities  Reauired  for  Current  Increment 

SoS  Synchronization 

OV-1  Hiah-Level  Graohic 

OV-5  Operational  Activitv  Model 

OV-2  Operational  Node  Connectivity  Description 

OV-4  Oraanizational  Relationship  Chart 

SV-1  Svstem/SoS  Interface  Description 

SV-5  Operational  Activitv  to  Svstem/SoS  Function 
Traceability  Matrix 

SV-4  Svstem/SoS  Functionality  Description 
SV-6  Svstem/SoS  Data  Exchange  Matrix 


0.2.2  Analyze  Blue 

□ 

Blue  Forces 

□ 

Blue  Actions 

□ 

Blue  Interactions 

0.2.5  Compose  Joint  Operational  Context 
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CTM  0.2. 3/4:  Analyze 
Environment/Analyze  Threat 


Recommended  Input  Info/Sources 

Products 

Example 

0.2  Develop  JOC-T  i 

Capability/SoS  Description 

I Q 1 

Joint  Operational  Context  for  Test  f  JOC-T)  | 

^  0.2,3  Analyze  Environment  i 

Physical  Environment  Conditions 

□  1 

Physical  Environment 

Civil  Environment  Conditions 

i 

□  1 

Civil  Environment 

Threat  and  Ooerationai  Environment 

< 

'  ^  0.2.4  Analvze  Threat  \ 

Analytical  Baseline  (DPS.  MSFD,  FYAB)  | 

a  1 

Threat  Forces  (SoS  Threat 
Assessment) 

Ooerationai  Overview 

□ 

Threat  Actions  (SoS  Threat 
Assessment) 

Threat  and  Operational  Environment 

□ 

Threat  Interactions  (SoS  Threat 
Assessment) 

Threat  Summary 

□  1 

Threat  Hioh-Level  Graohic  (TOV-1)  \ 

J.  J  ( 

Threat  Conditions 

Svstem/SoS  Threat  Assessment 

n 

0.2.5  Compose  Joint  Operational  Context 
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4.  Manage  Test 
Execution 


Joint 

Capability 

Evaluation 

(JCE) 


Event 

Management 

Plan 


JTEM  Capability  Test  Methodology 

(CTM)  v2.0 


6  Steps 
14 JTEM 
Processes 


1.  Characterize  Test 


Program 

Introduction 

Document 

(PID) 


Statement 

of 

Capabilities 

(SOC) 


Capability 
Subset 
Focus 


Develop  Test  Concept 
Refine  Evaluation  Strategy 
Technical  Assessment 


0.  Develop 
T&E  Strategy 


T&E 

T&E 

Strategy 

Master 

(TES) 

Plan 

(TEMP) 

Develop  Capability/SoS 
Description 

Develop  Joint  Operational 
Context  for  Test  (JOC-T) 
Develop  Evaluation  Strategy 
Develop/Refine  Capability 
Crosswalk 


2.  Plan  Test 


Test 


Plan 


Develop  Test  Design 
Perform  LVC  Distributed 
Environment  Analysis 
Develop  Test  Plan 


Test 

Concept  *. 

A 

Joint  Operational 
Context  for 


Test 

Data 

( 


Capability 

Set 

Focus 


5.  Evaluate  Capability 


Integrated 
*  Vignettes 

LVC 

Distributed 

Environment 

Design 

Joint  Mission 
Environment 

Test  Control  & 
>.  Monitoring 


3.  Implement 
LVC-DE 


System 

Design 

Document 

(SDD) 


Design  LVC  Distributed 
Environment  Configuration 
Integrate  LVC  Distributed 
Environment 


LVC  -  Live,  Virtual,  Constructive 
LVC-DE  -  Live,  Virtual,  Constructive 
Distributed  Environment 
SoS  -  System  of  Systems 


Event 

Focus 


Analyze  Data 

Evaluate  SoS  Performance  & 
Joint  Mission  Effectiveness 
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CTM  1.4:  Technical  Assessment 


CTM  1.1  Develop  Test  ConceptC  /  P  /  T| 


CTM. 1.1.1 

CTM.  1.1.2 

CTM. 1.1. 3 

Establish 

Overall 

Test  Goal 

Establish  Test 
Objectives 

Develop  Test 
Approach 

CTM. 1.2.1 


Establish 
Evaluation 
Basis  of 
Estimate 

CTM.1.2.2 


CTM  1.2 

Refine  Evaluation 
Strategy 


Develop  Test 
Scenario 


CTM.1.2.3 


Identify 
Additional 
Characterize 
Test  Modeling 
Requirements 


C/P/T 


CTM. 1.3 

P  /  T  /  R  /  DRCT 

Programmatic 

Assessment 

1 

1 

y~ 

-  -  AT 

CTM  1.4  Technical  Assessment 

■ 


CTM.1.4.3 

Analysis  of 
Alternatives 


CTM.  1.4.1 

CTM. 1.4.2 

Develop  Initial 
LVC-DE 
Operational 
Design 
Description 

Develop 

LVC-DE 

Alternatives 

P  /  T  /  R  /  DRCT 


CTM. 1.4.4 

Identify  New 
Development 
&  Integration 
Requirements 


CTM  1 

Characterize  Test 
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CTM  1.4:  Technical  Assessment 


Recommended  Input  Info/Sources 


4  Technical  Assessment  i 

1  □  4  Technical  Assessment  i 

l 

w\  1.4.1  Develon  Initial  LVC-DE  Operational  Design  1 

JME  Foundation  Model 

i  □  i  LVC-DE  OV-1  i  B  ( 

Previous  LVC-DE  Estimates 

i  □  i  LVC-DE  OV-4  i 

Proqram  Introduction  Document 

i  Ui  LVC-DE  OV-5  i  C  i 

i 

1.4.2  Develop  LVC-DE  Alternatives  ( 

LVC-DE  OV-1 

i  □  i  Distributed  Range  Capabilities  Matrix  «4| A k  A  i 

LVC-DE  OV-4 

i  □  i  LVC-DE  SV-1  i  D  i 

LVC-DE  OV-5 

i  □  t  LVC-DE  SV-4a  ^  4  j  E 

a  i  lvc-de  sv-4b  — e 

JQ  i  LVC-DE  SV-6  i  F 

□  ^  LVC-DE  SV-IOc  (  G  < 

Distributed  Ranae  Capabilities  Matrix 

""  1.4.3  Analysis  of  Alternatives 
i  □  i  High  Level  Schedule 

LVC-DE  SV-1 

LVC-DE  SV-4a 

i  □  i  Test  Resource  Estimate 
i  Q  ^  Technical  Recommendation 

LVC-DE  SV-4b 

i  □  4  Statement  of  Capabilities 

LVC-DE  SV-6 

LVC-DE  SV-IOc 

I 

LVC-DE  SV-1 

1 .4.4  Identify  New  Development  and  Integration  ( 

,-j  |  New  LVC-DE  Enterprise  Development 

1  Reauirements 

i 

LVC-DE  SV-4a  i 

LVC-DE  SV-4b  \ 

LVC-DE  SV-6  { 

LVC-DE  SV-IOc  i 

Technical  Recommendation  i 
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4.  Manage  Test 
Execution 


Joint 

Capability 

Evaluation 

(JCE) 


Event 

Management 

Plan 


JTEM  Capability  Test  Methodology 

(CTM)  v2.0 


6  Steps 
14 JTEM 
Processes 


0.  Develop 
T&E  Strategy 


T&E 

T&E 

Strategy 

Master 

(TES) 

Plan 

(TEMP) 

Develop  Capability/SoS 
Description 

Develop  Joint  Operational 
Context  for  Test  (JOC-T) 
Develop  Evaluation  Strategy 
Develop/Refine  Capability 
Crosswalk 


1. 

Characterize  Test 

Program 

Statement 

Introduction 

of 

Document 

Capabilities 

(PID) 

(SOC) 

•  Develop  Test  Concept 

•  Refine  Evaluation  Strategy 

•  Technical  Assessment 

Capability 
Subset 
Focus 


2.  Plan  Test 


Test 


Plan 


Test 

Concept  *. 

A 

Joint  Operational 
Context  for 


Test 

Data 

( 


Capability 

Set 

Focus 


5.  Evaluate  Capability 


Develop  Test  Design 
Perform  LVC  Distributed 
Environment  Analysis 
Develop  Test  Plan 

• 

Integrated 
.*  Vignettes 

/  LVC 

Distributed 
Environment 
Design 


Joint  Mission 
Environment 

Test  Control  & 
Monitoring 


3.  Implement 
LVC-DE 


System 

Design 

Document 

(SDD) 


Design  LVC  Distributed 
Environment  Configuration 
Integrate  LVC  Distributed 
Environment 


LVC  -  Live,  Virtual,  Constructive 
LVC-DE  -  Live,  Virtual,  Constructive 
Distributed  Environment 
SoS  -  System  of  Systems 


Event 

Focus 


Analyze  Data 

Evaluate  SoS  Performance  & 
Joint  Mission  Effectiveness 
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CTM  2.2:  Perform 
LVC-DE  Analysis 
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CTM  2.2:  Perform 
LVC-DE  Analysis 


Recommended  Input  Info/Sources 


Products 


2,2  Perform  LVC-DE  Analysis 
□  4  LVC-DE  Analysis 


L 


j  JME  Foundation  Model 
d  Data  Analysis  Plan 
jTest  Conceot 

System  and  Joint  Mission  Evaluation  Strategy 
LVC-DE  OV-1  High  Level  Operational  Concept 


LVC-DE  OV-1 
LVC-DE  OV-2 
LVC-DE  OV-4 
LVC-DE  OV-6c 
LVC-DE  OV-7 
LVC-DE  SV-4a  (Initial) 
LVC-DE  SV-4b  (Initial) 
LVC-DE  SV-6  (Initial) 
LVC-DE  OV-IOc  (Initial) 


2.2,1  Develop  Detailed  LVC-DE  Operational 
Description 

□  i  LVC-DE  OV-1  (Refined) 

□  |  LVC-DE  OV-2 
Q  I  LVC-DE  OV-4  (Refined) 

□  (  LVC-DE  OV-5  (Refined) 

□  (  LVC-DE  OV-6c 
Qi  LVC-DE  OV-7 

^2.2.2  Develop  LVC-DE  System  Functional 
Description 

□  {  LVC-DE  SV-4a  (Refined) 

□  (  LVC-DE  SV-4b  (Refined) 

LVC-DE  SV-6  (Refined) 

LVC-DE  OV-IOc  (Refined) 
Environmental  Specifications 
Refined  Configuration  Mgt  Plan 
Refined  V&V  Plan 


Example 


B 

C 


< 


) 


□ 

□ 


! 

a! 

,  Di  r 
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4.  Manage  Test 
Execution 


Joint 

Capability 

Evaluation 

(JCE) 


Event 

Management 

Plan 


JTEM  Capability  Test  Methodology 

(CTM)  v2.0 


6  Steps 
14 JTEM 
Processes 


0.  Develop 
T&E  Strategy 


T&E 

T&E 

Strategy 

Master 

(TES) 

Plan 

(TEMP) 

Develop  Capability/SoS 
Description 

Develop  Joint  Operational 
Context  for  Test  (JOC-T) 
Develop  Evaluation  Strategy 
Develop/Refine  Capability 
Crosswalk 


1. 

Characterize  Test 

Program 

Statement 

Introduction 

of 

Document 

Capabilities 

(PID) 

(SOC) 

•  Develop  Test  Concept 

•  Refine  Evaluation  Strategy 

•  Technical  Assessment 

Capability 
Subset 
Focus 


2.  Plan  Test 


Test 


Plan 


Develop  Test  Design 
Perform  LVC  Distributed 
Environment  Analysis 
Develop  Test  Plan 


Test 

Concept  *. 

A 

Joint  Operational 
Context  for 


Test 

Data 

( 


Capability 

Set 

Focus 


5.  Evaluate  Capability 


.*  Integrated 
Vignettes 

LVC 

Distributed 

Environment 

Design 

Joint  Missioi 
Environmen. 

Test  Control  i 
Monitoring 


3.  Implement 
LVC-DE 


System 

Design 

Document 

(SDD) 


Design  LVC  Distributed 
Environment  Configuration 
Integrate  LVC  Distributed 
Environment 


LVC  -  Live,  Virtual,  Constructive 
LVC-DE  -  Live,  Virtual,  Constructive 
Distributed  Environment 
SoS  -  System  of  Systems 


Event 

Focus 


Analyze  Data 

Evaluate  SoS  Performance  & 
Joint  Mission  Effectiveness 


item@ite.osd.mil 


CTM3.1:  Design  LVC-DE 
Configuration 


CTM  3.1  Design  LVC-DE  Configuration 

T  /  R / DRCT / O 


CTM. 3. 1.1 


Refine  JME 
LVC 

Foundation 

Model 


CTM. 3. 1.3 


CTM. 3. 1.2 


Perform  JME 
>  LVC  Logical 
Design 


Characterize 

Test 

Infrastructure 

Performance 


CTM. 3. 1.4 


Perform  JME 
LVC  Physical 
Design 


j|  CTM. 3.2  ^ 

Build/Configure 
LVC-DE 
Components 


R/O 


CTM  3 

Implement  LVC-DE 


CTM. 3.4.1 


CTM. 3. 3 

Encode 

Vignettes 

R/O 

CTM  3.4  Integrate  LVC-DE 


CTM. 3.4.2 


Integrate 

Local 

Systems 


Verify 

Distributed 

Systems 


CTM. 3. 4.3 


Pilot 

Check-Out 


T  /  R / DRCT / O 


i _ 
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CTM3.1:  Design  LVC-DE 
Configuration 


Recommended  Input  Info/Sources  Products 


3.1  Design  LVC-DE  Configuration 
[  □  i  LVC-DE  Configuration 


V  3. 1. 1  Refine  JME  LVC-DE  Foundation  Model  (JFM) 

Enterprise  JME  Foundation  Model  with  AV-2,  TV-1 

□  |  JME  Foundation  Model  (JFM)  Design 

LVC-DE  OV-1.  2.  4.  5,  6c,  7,  SV-4,  6, 10c 

i 

□  4  AV-2  (refined)  < 

□  4  TV-1  frefined)flHHHHHHHi 

□  4  LVC-DE  OV-1  (refined) 

a  4  LVC-DE  SV-4a/b  (refined) 

□  I J  LVC-DE  SV-6,  (refined) 

□  (  LVC-DE  SV-1  Oc  (refined) 

”v  3.1.2  Perform  JME  LVC-DE  Logical  Design 
JME  Foundation  Model  Design  I  □  9  JME  LVC-DE  Logical  Design 


LVC-DE  OV-1, 2, 4,  5,  6c.  7.  SV-4a/b,  6, 10c 

Data  Analysis  Plan 

Vignettes 

LVC-DE  Test  Accroach  Description 
Components  and  Interface  V&V 
Initial  V&V  Plan 

Initial  Configuration  Management  Plan 


4  □  4  LVC-DE  OV-1  (Logical) 

4  □  4  LVC-DE  OV-2  (Loaical) 

(  a  4  LVC-DE  OV-4  (Logical) 

4  □  4  LVC-DE  OV-5  (Logical) 

4  □  4l  LVC-DE  OV-6c  (Logical) 

4  a  4  LVC-DE  OV-7  (Logical) 

4  □  4  LVC-DE  SV-4a/b  (Loaical) 

□  4  LVC-DE  SV-6  (Logical) 

□  4  LVC-DE  SV-1 0c  (Logical) 

□  4  LVC-DE  AV-2  (Loaical) 

a  4  LVC-DE  TV-1  (Loaical) 

Q4  V&V  Plan 

□  |  LVC-DE  Design  Gap  Analysis 
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CTM3.1:  Design  LVC-DE 
Configuration 


Recommended  Input  Info/Sources 

Products 

3.1  Design  LVC-DE  Configuration  4 

i 

1  Ui  LVC-DE  Configuration  4 

3.1.3  Characterize  Test  infrastructure  Performance 

□  |  JME  Infrastructure  Characterization  Plan 

□  (  JME  Infrastructure  Characterization  ReDort 


4 


JME  LVC-DE  Loqical  Design  with  LVC-DE  OV-1, 2  4^1 
5. 6c,  7.  SV-4a/b.  6. 10c,  AV-2,  TV-1  (LDD) 

3. 1.4  Perform  JME  LVC-DE  Physical  Desion  < 

□  j  JME  LVC-DE  Physical  Design 

JME  Infrastructure  Characterization  Plan 

JME  Infrastructure  Characterization  Report 

□  (  LVC-DE  OV-1  (Physical) 

J2i  LVC-DE  OV-2  (Physical)  < 

Plan  M 

JO  i A  LVC-DE  OV-4  (Physical)  ' 

LVC-DE  Design  Gap  Analysis 

Repository  of  LVC  Capabilities  1 

□  4  LVC-DE  OV-5  (Physical) 

Ui  LVC-DE  OV-6c  (Physical) 

jQ  iA  LVC-DE  OV-7  (Physical) 

JQ  i  LVC-DE,  SV-4a/b  (Physical)  < 

JG  (  LVC-DE,  SV-6  (Physical) 

□  (  LVC-DE,  SV-1  Oc  (Physical)  ' 

Qi  LVC-DE  AV-2  (Physical)^ 

LVC-DE  TV-1  (Physical) 

□  |  Configuration  Management  Plan 
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Instantiated  JME 


Real  Time/Post-Test 
Processing 


Infrastructure: 
Network,  Middleware 


Non 
Real-Time 
Data 
Transfer 


Resource 
Data  Loggers 

Operational  Capability 
Data  Loggers 


LVC  systems/systems  of  systems 
representing  warfighting  capabilities 


Physical  Design  configuration  includes  systems  in 
Operational,  Test,  and  Data  layers 
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CTM  3.4:  Integrate  LVC-DE 


CTM  3.1  Design  LVC-DE  Configuration 

T  /  R / DRCT / O 


CTM. 3. 1.1 


Refine  JME 
LVC 

Foundation 

Model 


CTM. 3. 1.3 


CTM. 3. 1.2 


Perform  JME 
►  LVC  Logical 
Design 


Characterize 

Test 

Infrastructure 

Performance 


CTM. 3. 1.4 


Perform  JME 
LVC  Physical 
Design 


CTM.  3. 2 


Build/Configure 

LVC-DE 

Components 


R/O 


CTM. 3. 3 

V 

Encode 

Vignettes 

i 

R/O 

CTM  3 

Implement  LVC-DE 


CTM  3.4  Integrate  LVC-DE 


CTM. 3.4.1 


CTM. 3.4.2 


Integrate 

Local 

Systems 


Verify 

Distributed 

Systems 


CTM. 3. 4.3 


Pilot 

Check-Out 


T  /  R / DRCT / O 


I 

I 

r 
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CTM  3.4:  Integrate  LVC-DE 


Recommended  Input  Info/Sources 

Products 

3,1  Perform  LVC-DE  Analysis 

I  □  i  LVC-DE  Analysis  % 

L 

3.4. 1  tniearate  Local  Svstems 

4 

LVC-DE  Comoonents 

ll  r 

(  □(  Local  Confiauration  Report 

4 

Confiauration  Manaoement  PI; 
V&V  Plan  <!■■■■■■■■ 

’ 

4 

4 

Intearated  JME  Verification 
Documentation 

i 

3.4.2  Verify  Distributed  Svstems 

4 

Local  Confiauration  Report 

t  3i  LVC-DE  Confiauration  Report 

4 

i  □<  LVC-DE  Technical  Baseline 

4 

L> 

3,4.3  Pilot  Check-Out 

Vianette  Databases 

1  □!  Ooerat'ona*  JME  Validation 

1  1  Documentation 

J 

LVC-DE  Confiauration  Reoort 

i  Joint  Mission  Environment 

] 

LVC-DE  Technical  Baseline 

(  Q(  JME  Check-Out  Reoort 

| 
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Overview 


•  JTEM  Problem  Statement 

•  Capability  Test  Methodology  (CTM) 

•  CTM  Systems  Engineering  Thread 

•  Summary 
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Summary 


•  JTEM  mission  is  to  develop  methods  and 
processes  (M&P)  for  realistic  testing  in  a  live, 
virtual,  constructive  distributed  environment 
(LVC-DE) 

•  CTM  systems  engineering  process  provides  an 
effective  building  block  approach  to  JME 
development  -  “Design  Once  -  Use  Many” 

-  JFM 

-  Logical  Design 

-  Physical  Design 


item@ite.osd.mil 
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The  Role  of  Chaos  and  Complexity  in 
Systems  Development  s 

v  w.  4%  *.  aL5ifc 

Robert  J.  Monson,  Ph.D. 

Lockheed  Martin  MS2  Tactical  Systems 

Eagan,  Minnesota 


Chaos  and  Complexity 

♦  The  Bak  Sandpile  model 

Defines  the  behavior  of  a  simple  system 

Representative  of  many  physical  and 
organizational  systems 

Provides  insight  into  an  appropriate 
method  to  plan  and  manage  systems 


Why  do  we  care? 


Work 

Level 


Time  -> 


How  does  the  model  work? 

*  Complex  Systems  are  frequently 
governed  by  simple  rules 

1.  Add  1  item  randomly 
to  any  pile 

2.  If  any  pile  >=  4  items 
distribute  4  items 


Number  of  Occurences 


Typical  Results 


25X25  Sandpile  Model 


- 

< 

► 

< 

► 

4 
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4 

► 

◄ 

►  ◄ 
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► 

.  ▲ 
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W 

- * - 

1  10  100 

Number  of  Avalanches 


The  Sandpile  Model 

*  Examples  utilizing  a  3  X  3  matrix 

Previous  example 

*  Larger  examples  do  not  exhibit  such 
dramatic  edge  effect 

25  X  25  model  used  most  commonly 

Use  simulation  to  provide  behavior 
information 


--  one  particle 
green  box  --  two  particles 
blue  box  --  three  particles 
red  box  --  four  particles,  critical  (unstable)  state 


3X3  matrix  100  points 


25  X  25  matrix,  2000  pts 


25  X  25,  2M  points,  1875  Normalized 


7000  n 


lo  O)  co  in  a)  cor^-T-Loaicor^T-mcDcor^T-  mcDcor^ 
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25  X  25,  2M  points,  1875  Normalized 
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—  Average 
■■ —  Maximum 
Productivity 


-3000  J 


25  X  25,  2M  points,  1875  Normalized 


-+ —  Average 
-■ —  Maximum 


Productivity 


25  X  25,  2M  points,  1875  Normalized 


Some  Examples 

♦Physical  Models 

♦Traffic  Patterns 

♦Complex  Interactions  in 
Organizational  Systems  / 
Systems  Development 


Physical  Models 


♦  Fish  Schooling 

♦  Oslo  Experiment 

Rice  grains 
between  sheets 
of  glass 

♦  Avalanches 
monitored 


T raffic  Patterns 


Microsimulation  of  road  traffic  with  a  time-continuous  model 


Start 

1:  Ring  Road 

3:Laneclosing 

Stop 

2:0n-Ramp 

4:Uphill  Grade 

T raffic  Patterns 


Organizational  Systems 

#  Predictability  of  complex  systems  is  effective 
in  a  generalized  sense 

I  cannot  know  when  and  where  earthquakes  will 
occur,  but  I  can  know  approximately  how  many  to 
expect  and  typical  magnitudes 

Overall  I  will  have  a  good  idea  what  energy  will  be 
imparted  by  the  earthquakes 

This  is  good  enough  to  know  how  to  design 
structures  for  the  region 

*  Systems  Design  requires  predictability  in 
order  to  achieve  plans  and  projections 


Systems  Design 

*  To  increase  probability  of  success,  we  need 
to  dramatically  increase  operational 
predictability 

#  Scheduling  work  with  a  consideration  for  75% 
efficiency  provides  this  added  predictability 

Since  we  do  not  know  what  specific  disturbances 
will  occur 

We  do  not  know  when  they  will  occur  or  what 
magnitude  they  will  be 

But  we  know  that  on  average  that  25%  of  our  time 
will  be  consumed  by  them 


Conclusions 


*  A  complex  system  will  organize  itself  into  a  critical  (or 
unstable)  state 

*  We  know  that  a  certain  amount  of  disturbances  and 
resultant  avalanches  within  our  Systems 
Development  is  unavoidable 

*  We  don’t  know  specifics,  but  we  know  25%  of  our 
time  will  be  consumed  by  interdependencies  in  the 
system 

*  We  can  increase  our  probability  of  success  by 
planning  personnel  at  75%  capacity,  which  should  be 
created  as  our  maximum  productivity 

*  This  purposeful  detuning  of  the  system  results  in 
fewer  catastrophes  with  less  catastrophic  Systems 
Development  results 
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Objective 


■  Refocus  programs  back  to  basic  objectives  of  Systems  Engineering 
execution  including  oversight  of  product  developmental  life  cycle 

-  Requirements,  design,  implementation,  test,  delivery,  product  feedback  and 
sustainment 


■  Identify  methods  of  simplifying  and  presenting  key  domain 
knowledge  (need  to  know)  to  the  engineer 

-  Processes,  procedures,  and  technical 


■  Provide  simplified  approaches  to  improve  communication  and 
better  manage  products  and  teams 

-  Use  of  web,  database  tools  and  improvement  focals 

■  Provide  ability  to  better  understand  and  manage 
products  in  an  age  of  sometimes  overwhelming 
conditions 

-  Reduce  the  apparent  bottleneck  caused  by  engineering 
teams  interpreting  the  overlapping  requirements  and 
mandates 


Mgt 

r'^rytates 


S/W 

Tools 


Policies 
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Engineer 


Traditional  Systems  Engineering 


PROCESS  OUTPUT 


Source:  Systems  Engineering  Fundamentals  -  DOD  Publication,  Defense  Acquisition  University  Press 
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Traditional  Systems  Engineering 


■  Fundamental  Systems  Engineering  Activities 

-  Requirements  Analysis 

-  Functional  Analysis  and  Allocation 

-  Design  Synthesis 

■  All  balanced  by  techniques  and  tools  called  System  Analysis  and 
Control 

-  Track  Decisions  and  Requirements 

-  Manage  Interfaces 

-  Manage  Risks 

-  Track  Cost  and  Schedule 

-  Track  Technical  Performance 

-  Verify  Requirements 

-  Review  and  audit  progress 
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Can  we  improve  Systems  Engineering? 


■  Processes,  Procedures  and  Technical  Information 

-  Decrease  excess  of  supporting  documentation  including 
variations  of  same? 


■  SEI  CMM®,  SEI  CMMI®,  corporate,  program,  team,  etc 

-  Legacy  programs  struggle? 

■  Baseline  to  one  set,  then  an  “improved”  set  is  flowed  down 
(sometimes  before  the  initial  baseline  is  completed) 

-  Identify  specific  information  related  to  engineering  role? 

■  Easy  to  get  lost  and  confused 

■  Systems  Engineering  Oversight 

-  Provide  oversight  during  code/build  to  decrease  chances  of 
major  rework  down  the  road? 

-  Evaluate  metrics  at  developmental  stages  and  post  delivery? 

■  Build  upon  successful  program  practices  and  lessons  learned 

■  Continuous  improvement  j 

-  Utilize  Improvement  Councils  with  dedicated  focals? 
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Management 

Mandates 


Previous  Assessment  Findings 


■  Quick  Assessment  Guidelines 

-  Begin  with  quick  assessment  of  group  developmental  status 

-  Identify  common  and  unique  enterprise  software  tools 

-  Identify  artifacts,  processes,  procedures  and  supporting  documentation 

-  Identify  all  change  boards  and  other  review  boards 

-  Identify  methods  for  group  communication  and  status 

■  Results  of  Evaluation 

-  Determined  that  many  processes,  procedures,  and  documentation  were 
already  in  use  accessible  via  program  only 

-  Programs  were  collecting  some  information  (give  credit  where  credit  is  due) 

-  Included  common  and  unique  tools  such  as  Finance/Budgeting,  Earned 
Value  System,  Risk  Tracking,  Quality  and  Selloff  documentation, 
Requirements  tracking,  Change  Process/CCB,  and  some  levels  of  metrics 

-  Big  picture  of  program  not  always  apparent  to  team  members 
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Focus  on  Following  Standard  Work  Flow 


■  Engineering  development  should  follow  a  basic  work  flow 

■  Problems  occur  when  basic  development  steps  are 
marginalized,  minimized,  or  omitted 


r 


Sustainment  support  and 
Capturing  product  upgrades 


£ 


Configuration 
Management, 
delivery  and 
continuous  improvement 


Testing,  verification  and  release. 


resting,  \ 
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fe  hi  m  wm  mm  &  mm 


Design  Synthesis 

-code/build 

-oversight 


Problem  disposition 
and  upgrades 


A 


Change 

Board 

activity 


New 

development 
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Work  Flow  Visualization 


■  Provides  the  stakeholders  with  complete  color  coded  work  flow  of 
both  new  products  and  sustainment  of  existing  products 

■  Visually  enhances  ability  of  the  stakeholders  to  better  understand 
dynamics  of  how  to  improve  systems  engineering  execution  and 
business  discipline  knowledge  management 
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Work  Flow  Visualization  (cont) 


■  Legend  provides  color  coded  element  identifiers 

■  Standard  tools  -  lists  web-based  methods  for  maintaining 
same  information  gathering  throughout  the  organization 


Legend 


Standard  Tools 

Web  Portal 

-  eGuidance 

-  Standard  Meeting  Agenda 

-  Status  Roll  Up 

-  Peer  Review 

-  Support  Center 


sri 


•#> 


■  +  m  m  m  s 


•©©•© 


Copyright©  2008  Boeing.  All  rights  reserved. 


Work  Flow  Visualization  (cont) 


■  Start  for  new 
development  Section 

-  Entry  point  for  new 
business 


Legend 


V 


Standard  Tools 

Web  Portal 

-  eGuidance 

-  Standard  Meeting  Agenda 

-  Status  Roll  Up 

-  Peer  Review 

-  Support  Center 


/ 
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Risks  -ri 


Business  Friifiusd 


Work  Flow  Visualization  (cont) 


■  Sustainment  support  and  capturing  product  upgrades 

-  Represents  methodology  for  acquiring  follow-on  business 


Internal  Cusomere 
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-  eGuidance 

-  Standard  Meeting  Agenda 

-  Status  Roll  Up 

-  Peer  Review 

-  Support  Center 


j 


Copyright©  2008  Boeing.  All  rights  reserved. 


Work  Flow  Visualization  (cont) 


■  Section  addresses  support  center,  problem  disposition  and 
upgrade  funding 


Work  Flow  Visualization  (cont) 


■  Section  for  Change 
Board  activity 


Legend 


V 


Standard  Tools 

Web  Portal 

-  eGuidance 

-  Standard  Meeting  Agenda 

-  Status  Roll  Up 

-  Peer  Review 

-  Support  Center 


/ 
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Work  Flow  Visualization  (cont) 


■  Design  Synthesis  -  code/build  oversight 


Legend 


V 


Standard  Tools 

Web  Portal 

-  eGuidance 

-  Standard  Meeting  Agenda 

-  Status  Roll  Up 

-  Peer  Review 

-  Support  Center 
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Work  Flow  Visualization  (cont) 


■Testing,  verification  and  release 


Verification 

TestReq 

Traceability 


Release 
Board  ATP 


Legend 


V 


Standard  Tools 

Web  Portal 

-  eGuidance 

-  Standard  Meeting  Agenda 

-  Status  Roll  Up 

-  Peer  Review 

-  Support  Center 
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Work  Flow  Visualization  (cont) 


■  Configuration  Management,  delivery  and  continuous 
improvement 


■  Improvement  Council 


Notify  Customer 


Delivery 

A 


Legend 


V 


Standard  Tools 

Web  Portal 

-  eGuidance 

-  Standard  Meeting  Agenda 

-  Status  Roll  Up 

-  Peer  Review 

-  Support  Center 
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► 


Configuration 
Management 
Audit  (QA) 


Work  Flow  Visualization  Benefits 


■  Identifies  major  steps  in  development  that  will  remain  during 
organizational  process  change  activity 

-  Engineer  better  informed  as  to  what  his  or  her  role  is  for  product 
development 

-  Influence  to  product  delivery 

■  Associated  processes  may  change,  but  work  flow  stays  consistent 

-  Minor  adjustments  made  for  that  role  for  that  task 

■  Communication  across  specialties  improved 

-  Work  flow  task 

■  Importance  of  work  flow  task  provides  increased  importance  on  work 
product  artifact,  at  that  stage 

-  Improve  peer  review  effectiveness 

-  Decreases  chance  of  out  of  phase  defects 

-  Increases  chance  of  in  phase  defects  found 
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Knowledge  Management 


■  Linking  and  sharing  of  related  information  between  business 
disciplines 

-  Improves  systems  engineering  influence  and  maturity 

-  Improves  oversight  of  quality 

-  Increases  timeliness  of  applicable  decision  making  processes 

-  Directs  engineer  to  key  “need  to  know”  information 

-  Protects  engineer  from  overwhelming  sensation  of  “nice  to  know” 
information 

-  Reduces  bottleneck 
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Knowledge  Management  (cont) 


■  Electronic  guidance  or  eGuidance 

■  Key  “need  to  know”  information  provided  by  a  web  based 
tool 

-  Procedures,  Processes,  and  tools  required  to  do  the  job 

-  E-Guidance  is  a  tool  designed 
to  provide  an  employee  relevant 
reference  information  regarding 
his/her  role  and  responsibilities 
within  the  organization  and 
current  assignment 

employee  focus  learning  of 
expeditious  manner 


necessary  tools,  procedures, 
and  product  documents  in  an 


-  Intent  of  e-Guidance  is  for  the 
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Standard  Tools  to  Consider 


■  Common  Web  Portal 

-  Meeting  Agenda 

-  Meeting  Minutes 

-  Status  with  applicable  roll  up  to  various  levels  of  leadership 

-  eGuidance 

-  Peer  Review 

-  Support  Center 
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Summary 


■  Challenge 

-  Implement  an  effective  method  of  improving  systems  engineering  execution  and 
knowledge  management  across  specialties 

-  Maintain  control  of  chaotic  situations  that  impact  base  lined  work  flow 

-  Insure  communication  of  activities  are  readily  available  up  and  down  the 
organizational  chain 

■  Solution 

-  Build  on  past  studies  and  lessons  learned  for  continuous 
improvement 

-  Develop  visualizations  of  major  business  work  flow  elements 

-  Map  the  employee  role  to  the  documentation  that  is  needed 

-  Develop  standard  meeting  agendas  that  represent  full  process 
compliance 

-  Utilize  the  latest  technology  to  lessen  the  bottle  neck  affect  of  key 
domain  technical  documentation  of  the  team  and  specific  roles 

■  Future  Benefits 

-  More  robust  program  managers 

-  Knowledge  builds  upon  knowledge 


Engineer 
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Management 

Mandates 
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Outline 


•  Introduction 

•  Objectives  of  SOA 

•  Advantages  &  Implementations  of  SOAs 

•  Objectives  of  Net-Centric  Warfare 

•  Implementations  of  Net-Centric  Warfare 

•  Common  Features 

•  Fundamental  Considerations 

•  Baseline  Architecture  Questions 

•  Conclusions 
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Introduction 


SOAs  provide  agility  by  giving  users: 

•  Open  &  interoperable  system  design 

•  A  structure  for  problem  &  requirement  resolution 

•  Common  best  practices  &  systems  engineering  techniques 

•  Consistency  across  the  industry 

•  A  vehicle  for  sharing  strategies  and  proven  approaches 
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Objectives  of  SOA 


SOA’s  principal  objectives  are  to  provide 

•  Application  reuse 

•  Fast  response  to  business  needs 
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Advantages  &  Implementations  of  SOA 


Interface  Data  Unit 
Interface  Control  Info 
Service  Data  Unit 
Service  Access  Point 
Protocol  Data  Unit 

Header 


Objectives  of  Net-Centric  Warfare 


Net-Centric  Warfare’s  Holy  Grails: 

•  Timeliness 

•  Availability 

•  Throughput 
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Implementations  of  NCW 


_ ip _ 

Asynchronous  Transfer  Mode  (ATM) 
_ SONET/SDH _ 

Interface  for  OTN,  G.709 
Optical  Fiber/OTN  (WDM) 
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Common  Features 


Both  SOAs  and  Net-Centric  Warfare  require 

•  Stable  Requirements 

•  Correlation  of  Disparate  Stakeholders 

•  Strong  Management 


8 


Fundamental  Considerations 


IP 
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OSI 

Layer 

SONET 
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ATM 

Layer 

ATM 

Sublyr 

Functionality 
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CS 
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SAR 

Segmentation  and  reassembly 
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ATM 

Flow  control 

Cell  header  generation  &  extraction 

Virtual  circuit  path  management 

3 

2 

Phys 

TC 

Sell  ra^e  '(Je^upll^  header,  Checksum,  Frame  generation, 

Packing  and  unpacking  cells  from  enclosing  envelope 

1 

1 

Phys 

PMD 

Bit  timing  and  physical  network  access 
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Baseline  Architecture  Questions 


•  Should  Architecture  Be  Software  Based? 

•  Is  an  Enterprise  Service  Bus  Appropriate? 

•  Should  the  SOA  Be  Implemented  By  a  Single 
Vendor/Integrator? 
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Conclusions 


•  The  SOA  can  either  compliment  or  impede  Net-Centric 
Principles 

•  Implementations  should  be  pursued  with  adequate 
prototyping  and  testing 
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Abbreviations 


•  AAL  -  ATM  Adaptation  Layer 

•  ATM  -  Asynchronous  Transfer  Mode 

•  CS  -  Convergence  Sublayer 

•  ICI  -  Interface  Control  Info 

•  IDU  -  Interface  Data  Unit 

•  IP  -  Internet  Protocol 

•  NCW  -  Net-Centric  Warfare 

•  OSI  -  Open  System  Interconnection 

•  OTN  -  Optical  Transport  Network 

•  PDU  -  Protocol  Data  Unit 

•  PMD  -  Physical  Medium  Dependent 

•  SAP  -  Service  Access  Point 

•  SAR  -  Segmentation  and  Reassembly 

•  SDH  -  Synchronous  Digital  Hierarchy 

•  SDU  -  Service  Data  Unit 

•  SOA  -  Service  Oriented  Architecture 

•  SONET  -  Synchronous  Optical  Network 

•  TC  -  Transmission  Convergence 

•  WDM  -  Wave  Division  Multiplexing 
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LOCKHEED  M  A  R  T  I  H  / 


Introduce  the  Problem 


No  consistent  definition  &  process: 

Results  of  a  Web  Search  “System  Integration  is 
the  successful  integration  of  a  new  technology 
into  the  system  by  analyzing  the  technology's 
system  effects  and  resolving  any  negative 
impacts  that  might  result  from  its  broader  use.” 

From  the  International  Council  on  System 
Engineering  (INCOSE)  web  site  -  “Integrate:  .  . 

.  Systems,  businesses  and  people  must  be 
integrated  so  that  they  interact  with  one 
another.  Integration  means  bringing  things 
together  so  they  work  as  a  whole. . .” 


October  2008 
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LOCKHEED  M  A  R  T  I  H  / 


Introduce  the  Problem  (cont’d) 


My  favorite  published  definition: 

Integration  is  defined  as  the  act  of  mating 
hardware  and/or  software  components, 
subsystems,  systems  or  elements  at  their 
respective  interfaces  and  verifying  the 
compatibility  and  proper  operation  of  the 
integrated  units. 

From  a  paper  entitled  “Integration  Challenges  of 
Complex  Systems”  written  by  Bill  Haskins  and  Jack 
Striegel  for  the  16th  Annual  INCOSE  International 
Symposium, 

No  complete  guidance  on  how  to  do  Integration 


October  2008 
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LOCKHEED  M  A  R  T  I  H  / 


Integration  throughout  the  Lifecycle 


Detailed  Design  &  Assenible/BuiUI  Hardware  &  Software  Products 


PDR/CDR 


Goal  l&V  Documents: 
Draft  at  SRR,  SDR  &  PDR 
Finals  at  CDR 


Operations  Environment 
TEMP  Driven 


System  &  SoS  Validation 
-  Proce  do  res/Re  ports 
Operational  Test  &  Evaluation 


SRR 


SDR 


C  ust  cm  e  r  Re  quir  em  e  nts 
Customer  Concept  of  Ops 
Cusiomer  Validation  Plane 


Cost, Schedule,  Risks 


Subsystem  Requirements 
Interface  Design  Documents 
Subsystem  Verification  Plans 


Subsystem  Integration 


Subsystem 


Subsystem  Verification 
-  Verification  Test 
Procedure/Report 
Subsystem  Integration 


Product  Specifications 
Product  Verification  Plans 
Product  Integration  Plans 


Product 


Pro  duel  Verification 
-  Ve rifie anon  Pro c/Report 
Product  Integration 
HW  ^  SW  Component/Unit  Test 


System 

Master  l&V  Plan  Driven 


System  Verification  Testing 
-  Verification  Test 
Procedure/ Re  port 
System  Integral  ion 


System  Requirements 
Operational  Scenarios 
System  Verification  Plan 
_ Interface  Control  Doc's 

integral  ion  Threads 
System  Integral  ion  Plan 


October  2008 
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LOCKHEED  M  A  R  T  I  H  / 


Integration:  Techniques,  Methods  &  Tools 


Two  Techniques: 

Non-lncremental*  (Big  Bang)  vs  Incremental* 

Incremental  is  the  way  to  go  for  most  systems 
and  large  applications 

■  Integrate/Build-Up  -  starting  small  and 
continually  increasing  capability/complexity 


*  References  for  Techniques  and  Methods 

Kit,  Edward.  1995.  Software  Testing  in  the  Real  World,  Addison-Wesley 
Myers,  Glenford.  1979.  The  Art  of  Software  Testing,  John  Wiley  &  Sons,  Inc. 


October  2008 
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LOCKHEED  M  A  R  T  I  H  / 


Integration:  Techniques,  Methods  &  Tools 

Three  Methodologies: 

Top-Down*,  Bottoms-Up*  &  Thread-Based 


October  2008 
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LOCKHEED  M  A  R  T  I  H  / 


Integration:  Techniques,  Methods  &  Tools 

Three  Methodologies:  _ m 

■  Top-Down  I — cb  cp — i 


October  2008 
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LOCKHEED  MARTIN 


Integration:  Techniques,  Methods  &  Tools 


Three  Methodologies: 
■  Bottom-Up 


Driver  E 
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LOCKHEED  M  A  R  T  I  H  / 


Integration:  Techniques,  Methods  &  Tools 

Three  Methodologies:  I  ^  ^ _ 

■  Thread-Based 

■  Experience  indicates  this  is  I - 1 - 

the  preferred  method  for  most  D 

large  complex  applications 

and  or  systems  , - ^ - , 


October  2008 
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LOCKHEED  M  A  R  T  I  H  / 


Integration  Support  Activities 

Interface  Matrices  (Interface  Coverage) 

Account  for  all  internal  &  external  interfaces 
Hardware/Software/System  Build  Plan 

Thread  based  and  negotiated  with  the  developers 
Dedicated  Integration  Laboratories 
Separate  from  Test  Laboratories 
Early  “ilities”  Checkout  during  integration  phases 

■  Stability 
Reliability 

■  Performance 

■  Capacity 


October  2008 
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LOCKHEED  M  A  R  T  I  H  / 


Wrap-Up 

Integration  requires  a  different  skill  set  than 
Testing. 

Lessons  learned  have  shown  that  Integration 
is  a  key  weakness  on  most  medium  to  large 
software  intensive  projects 

■  Perform  the  Top  Ten  Integration  steps  and 
you  will  have  a  robust  Integration  process 


October  2008 
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LOCKHEED  M  A  R  T  I  H  / 


Top  Ten  Integration  Steps 


1.  Document  the  Integration  and  Test  process 

2.  Hire  and  train  the  right  staff  for  the  role  of  Integrator 

3.  Review  and  analyze  requirements  to  ensure  testability  and  included 
requirements  to  ensure  visibility  into  system  data  while  it  is  operating 

4.  Ensure  all  interfaces  at  all  levels  of  the  architecture  have  been 
identified  and  are  implemented,  tested,  tracked,  and  statused 

5.  Identify  &  plan  other  testing  activities  to  start  during  the  integration  test 
conduct  phase  (i.e.  stability,  performance,  reliability,  etc) 

6.  Develop  and  maintain  a  Project  “Build  Plan” 

7.  Define  and  ensure  sufficient  Integration  and  Test  laboratories  available 

8.  Design  integration  tests  and  test  data  for  all  levels  of  the  architecture 

9.  Ensure  functional  testing  is  also  being  conducted  at  each  level  of  he 
architecture 

10.  Ensure  sufficient  simulation/stimulation  capabilities  are  available 


October  2008 
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Technology  Readiness 
Assessments  (TRAs)  for 
Systems  of  Systems  (SoS) 


2008  National  Defense  Industrial  Association 
11th  Annual  Systems  Engineering  Conference 

October  21, 2008 

Dr.  Jay  Mandelbaum 

Institute  for  Defense  Analyses 

4850  Mark  Center  Drive  -  Alexandria,  Virginia  22311-1882 


Outline 


•  Background 

*  Complexity  of  the  problem 

•  The  TRA  process  for  a  SoS 

-  Describing  the  SoS 

-  Identifying  the  SoS  environment(s)  and  interfaces 

-  Identifying  SoS  CTEs  and  their  associated 
relevant/operational  environments 

-  Conducting  the  SoS  TRA 

-  Documenting  and  coordinating  the  SoS  TRA 

*  SoS  TRA  updates 


What  is  a  TRA? 


Systematic,  metrics-based 
process  that  assesses  the 
maturity  of  Critical 
Technology  Elements  (CTEs) 

-  Uses  Technology  Readiness 
Levels  (TRLs)  as  the  metric 

Regulatory  information 
requirement  for  all 
acquisition  programs 

-  Submitted  to  DUSD(S&T)  for 
ACAT  ID  and  1AM  programs 


+  Not  a  risk  assessment 

=£  Not  a  design  review 

±  Does  not  address  system 
integration 


Why  is  a  TRA  Important?  (i  of  2) 


The  Milestone  Decision  Authority 
(MDA)  uses  the  information  to  support 
a  decision  to  initiate  a  program 

-  Trying  to  apply  immature  technologies 
has  led  to  technical,  schedule,  and  cost 
problems  during  systems  acquisition 

-  TRA  established  as  a  control  to  ensure 
that  critical  technologies  are  mature, 
based  on  what  has  been  accomplished 


•  Congressional  interest  j/ 

-  MDA  must  certify  to  Congress  that 
the  technology  in  programs  has 
been  demonstrated  in  a  relevant 
environment  at  program  initiation 

-  MDA  must  justify  any  waivers  for 
national  security  to  Congress 
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TRL  Overview 


*  Measures  technology  maturity 

*  Indicates  what  has  been  accomplished  in  the 
development  of  a  technology 

-  Theory,  laboratory,  field 

-  Relevant  environment,  operational 
environment 

-  Subscale,  full  scale 

-  Breadboard,  brassboard,  prototype 

-  Reduced  performance,  full 
performance 

*  Does  not  indicate  that  the  technology  is  right  for 
the  job  or  that  application  of  the  technology  will 
result  in  successful  development  of  the  system 


Critical  Technology  Element  (CTE)  Defined 


A  technology  element  is  “critical”  if  the  system 
being  acquired  depends  on  this  technology 
element  to  meet  operational  requirements  with 
acceptable  development  cost  and  schedule  and 

with  acceptable  production  and  operation  costs 
and  if  the  technology  element  or  its  application  is 
either  new  or  novel  or  in  an  area  that  poses  major 
technological  risk  during  detailed  design  or 
demonstration  or  provides  unprecedented 

functionality 

CTEs  may  be  hardware,  software, 
manufacturing,  or  life  cycle  related 
at  the  subsystem  or  component  level 
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Changes  Anticipated  for  New  Deskbook 


•  Reflects  new  DoDI  5000.02  and  other 
policy  changes 

•  Rigor  and  robustness  added  to 
processes  in  chapter  3 

•  Chapter  5  on  technology  maturity 
rewritten 

-  Early  evaluation  of  technology 
maturity 

-  Reflect  10  USC  2366a 

•  New  appendices 

-  Interfaces  with  S&T  community 

-  Space,  SoS,  and  ships 

The  following  material  is  preliminary. 
Feedback  is  welcome. 


Outline 


*  Background _ 

*  Complexity  of  the  problem 

*  The  TRA  process  for  a  SoS 

-  Describing  the  SoS 

-  Identifying  the  SoS  environment(s)  and  interfaces 

-  Identifying  SoS  CTEs  and  their  associated 
relevant/operational  environments 

-  Conducting  the  SoS  TRA 

-  Documenting  and  coordinating  the  SoS  TRA 

*  SoS  TRA  updates 
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Complexity  of  SoS  TRAs 


*  SoS  definition: 

a  set  or  arrangement  of  systems  that  results 
when  independent  and  useful  systems  are 
integrated  into  a  larger  system  that  delivers 
unique  capabilities 

•  Why  there  is  a  special  section  in  the 
Deskbook 

-  In  a  SoS,  individual  systems  are 
integrated  with  each  other  to  achieve  a 
capability 

-  An  individual  system’s  performance  is 
changed  by  its  linkage  to  other 
systems 

-  This  affects  both  CTE  identification 
and  CTE  assessment . . . 


Complexities  with  CTE  Identification 


CTEs  must  be  considered  tentative  prior  to 
completion  of  overall  SoS  engineering  and  then 
individual  system(s)  engineering 

-  SoS  operational/performance 
requirements  for  a  capability  are  not 
easily  allocated  to  individual 
systems  and  their  subsystems 

-  Some  of  the  interactions  among 
systems  are  not  predictable  in 
advance  and  the  individual  systems 
may  change  when  they  are  joined 
together 

-  The  allocation  of  SoS  operational 
and  performance  requirements  for  a 
capability  may  evolve  over  time 


Complexities  with  CTE  Assessment  in  a 
Relevant  SoS  Environment 


There  are  difficulties  in 
allocating  SoS  requirements  to 
associated  systems  or 
subsystems 

The  relevant  environment  may 
not  be  fully  understood  because 
other  systems  are  part  of  it: 

-  Modeling  and  simulation  may  not  be  adequate 

-  Test  and  evaluation  environments  may  not  be  fully 
understood 

-  System  performance  and  the  relationships  among 
systems  change  over  time 

-  Testing  all  permutations  is  not  possible 


Complexities  with  SoS  Management 


-  Each  of  these  systems  is 
often  managed  independently 

-  Control  of  resources  may  not 
be  collocated  with  those 
management  responsibilities 

-  The  associated  systems’ 
acquisition  activities  may  not 
be  on  the  same  time  line  as 
the  SoS  development  effort 


DoDI  5000.02  does  not  prescribe  an  overarching 
process  for  managing  SoS  acquisition  that  includes 
legacy  systems,  developmental  systems,  and 
system  modernization 
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Outline 


•  Background 

*  Complexity  of  the  problem 

•  The  TRA  process  for  a  SoS 

-  Describing  the  SoS 

-  Identifying  the  SoS  environment(s)  and  interfaces 

-  Identifying  SoS  CTEs  and  their  associated 
relevant/operational  environments 

-  Conducting  the  SoS  TRA 

-  Documenting  and  coordinating  the  SoS  TRA 

*  SoS  TRA  updates 
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Getting  Started 


Normal  TRA  best  practices  apply 

-  A  series  of  meetings  between  the 
program  office,  the  Component 
S&T  Executive  office,  and 
DUSD(S&T)  should  be  conducted 
to  determine  the  scope  and 
conduct  of  the  SoS  TRA 

-  Panel  members  should  be 
independent  of  the  program  and 
include  a  wider  range  of  relevant 
subject  matter  expertise 


14 


Describing  the  SoS 


Identify  the  boundaries  that  encompass  the  required  capability  to 
be  delivered 

Identify  SoS  spirals/blocks  or  other  expected  increments  and  their 
timeframes  including  spirals/blocks  of  specific  systems  of  the  SoS 

Identify  how  component  systems  must  be  modified  (and  new 
interfaces  developed)  to  be  integrated  into  the  SoS 

Identify  SoS  specific  operational  and 
performance  requirements,  including 
those  for  each  system  comprising  the  SoS 

Clearly  delineate  the  interface 
requirements,  externally 
controlled/managed  capabilities,  and  SoS 
dependencies  and  interdependencies, 
within  a  context  of  the  capabilities 
provided  and  operating  limits  of  the  SoS 
under  evaluation 


Identifying  the  SoS  Environment(s)  and 

Interfaces 

*  Focus  on  what  makes  the  SoS  environment  unique 

-  Consider  execution  time  or  data 
throughput  and  information 
exchange  requirements  to/from 
other  systems 

-  Include  information  assurance 
considerations 

-  Identify  functional 
dependencies  and  the 
technologies  that  enable  these 
functions 

*  If  no  documentation  exists  or  is  still  in 
development,  involve  the  SoS  end  users  or 
customers  for  determining  SoS  behavior(s) 
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Identifying  SoS  CTEs  and  their  Associated 
Relevant/Operational  Environments  <iof4> 


•  Identify  the  SoS  WBS,  ...  well  in  advance  of  the  CTE 
selection  and  systematically  examine  all  elements  of 
the  WBS  for  determining  CTEs 


When  conducting  a  TRA  for  the  SoS 

•  Include  all  CTEs  required  to  meet  SoS 
operational  requirements 

•  Include  SoS  unique  CTEs  and  system  unique 
CTEs  required  for  a  system  to  participate  in  the 
SoS  regardless  of  who  is  responsible  for 
funding  or  development 

•  Internal  and  external  dependencies  should  be 
treated  equally  and  all  associated  CTEs  should 
be  formally  assessed  in  the  SoS  TRA  against 
the  SoS  requirements 
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Identifying  SoS  CTEs  and  their  Associated 
Relevant/Operational  Environments  (2  of 4) 

-  When  conducting  a  TRA  for  a  system  that  is  part  of  the 
SoS 

•  Include  all  system  specific  technologies  that  meet  the  CTE 
criteria 

*  Assess  SoS  CTEs  that  are  in  the  system  undergoing  the 
TRA  even  if  they  are  not  system  specific  CTEs 

-  In  either  case,  take  into  account 
situations  where  a  capability  in  one 
system  is  dependent  on  a 
technology  in  another  system  for 
its  functionality 

-  Consider  any  TRA  completed  or 
being  conducted  on  a  system 
within  the  SoS  for  identification  of 
relevant  CTEs 


Identifying  SoS  CTEs  and  their  Associated 
Relevant/Operational  Environments  <3  of 4) 


Expand  upon  CTE  identification 

questions,  e.g., 

*  Does  the  technology  directly 
impact  an  operational 
requirement? 

-  Is  the  technology  contributing  to  a 
more  effective  performance  of  the 
SoS  in  development? 

-  Is  an  increase  or  change  in 
capability  being  required  from 
currently  fielded  systems? 

-  Is  the  technology  enabling  a  new 
concept  of  operation? 
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Identifying  SoS  CTEs  and  their  Associated 
Relevant/Operational  Environments  <4  of 4) 


Expand  upon  CTE  identification 

questions,  e.g.. 

Is  the  technology  new  or  novel  (or 

being  used  in  a  new  or  novel  way)? 

-  Is  this  technology  creating  new 
relationships  between  systems? 

-  Is  this  technology  dependent  upon  new 
relationships  between  systems? 

Has  the  technology  been  modified? 

-  Are  technologies  fielded  on  the  associated 
systems  being  modified  to  meet  new 
requirements  of  the  SoS? 


-  Are  current  technologies  dealing  with  the 
relationships  among  systems  being 
modified  for  the  SoS? 
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Conducting  the  SoS  TRA 


•  CTEs  in  a  component  system  whose 
environments  are  not  dependent  on  the  rest 
of  the  SoS  should  be  assessed  in  the  normal 
way 

•  The  Program  Manager  and/or  Chief  Engineer 
for  the  SoS  should  conduct  technology 
demonstrations/validations  for  the  SoS- 
related  CTEs 

•  The  program  should  provide  the  necessary 
documentation  to  the  Independent  Panel  to 
enable  independent  assessment  of  the  CTE 
performance  within  the  SoS 

•  The  Independent  Review  Panel  should 
determine  the  TRLs  for  each  SoS-related  CTE 


•  CTEs  for  future  spirals  are  not  expected  to  be 
TRL  6  at  Milestone/KDP  B 
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Documenting  and  Coordinating  the  SoS  TRA 


Reference  other  pertinent 
material 

Ensure  proper  coordination  is 
conducted  on  the  SoS 
technology  maturation  plans 
especially  between 
interdependent  acquisition 
development  efforts 

The  Component  Acquisition 
Executive  should 
communicate  the  TRA  results 
with  each  program 
management  echelon  which  is 
part  of  the  SoS 
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Outline 


•  Background 

*  Complexity  of  the  problem 

•  The  TRA  process  for  a  SoS 

-  Describing  the  SoS 

-  Identifying  the  SoS  environment(s)  and  interfaces 

-  Identifying  SoS  CTEs  and  their  associated 
relevant/operational  environments 

-  Conducting  the  SoS  TRA 

-  Documenting  and  coordinating  the  SoS  TRA 

*  SoS  TRA  updates 
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SoS  TRA  Updates 


•  Because  of  the  inherent  difficulties 
associated  with  SoS  TRAs  in  identifying 
CTEs  and  assessing  their  maturity  in  the 
relevant  environment,  it  is  possible  that 
even  a  rigorously  done  TRA  at  Milestone 
B  could  be  found,  in  hindsight,  to  be 
incomplete 

•  Update  at  SoS  CDR  and  initiation  of  new 
spiral  in  any  of  the  systems 


Milestone  B 

Milestone  C 

^  A 

AAA  A- 

—A 

A  A 

A 

WBS  So5 

Definition  Environ 
& 

Architecture 

CT  TRA 

Selec  nun 

Interim 

CTE 

Updates 

Final  CTE  Final 

Update  TRA 
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References  and  Resources 


Defense  Acquisition  Resource  Center  http://akss.dau.mil/darc/darc.html 

-  DoD  Directive  5000.1  (DoDD  5000.1),  The  Defense  Acquisition  System,  dated 
May  12,  2003 

-  DoD  Instruction  5000.2  (DoDI  5000.2),  Operation  of  the  Defense  Acquisition 
System,  dated  May  12,  2003 

-  Defense  Acquisition  Guidebook 


DAU  Continuous  Learning  Module  CLE021 

-  https://learn.dau.mil/html/clc/Clc.jsp  to  browse  it 
TRA  Deskbook 

-  http://www.defenselink.mil/ddre/doc/tra_deskbook_2005 

DDR&E 

-  Mr.  Jack  Taylor  jack.taylor@osd.mil 

Institute  for  Defense  Analyses 

-  Dr.  Dave  Sparrow  dsparrow@ida.org 

-  Dr.  Jay  Mandelbaum  jmandelb@ida.org 

-  Dr.  Michael  May  mmay@ida.org 


Modeling  &  Simulation 

in  the 


Test  &  Evaluation  Master  Plan 


NDIA  Systems  Engineering  Conference 

October  2008 

Michael  Truelove  -  SAIC  Support 
Office  of  the  Deputy  Under  Secretary  of  Defense  (A&T) 
Developmental  Test  &  Evaluation 
1550  Crystal  Drive,  Suite  1004 
Arlington,  VA  22202 
Phone:  703-412-3683 


Acquisition  M&S  Governance  Structure 


Industry 


SISO,  OMG,  etc. 
INCOSE 


SMAS,  SE  DSIG,  etc. 


INCOSE  MBSE  WG 


NDIA 

M&S  Committee 


DoD  Acquisition 

Chair:  Mr.  Gordon  Kranz 
ODUSD(A&T)/SSE 


NDIA 

Systems  &  Software 

SE  Division 

Engineering  Forum 

Acquisition 
M&S  Working  Group 


Chair:  Ms  Philomena  Zimmerman 
PM  FCS  (BCT) 

Associate  Director, 
Modeling  and  Simulation 


DoD  M&S 

Mr.  Di Petto 
Acquisition  Member: 
ODUSD(A&T)/SSE/DTE 


M&S  SC 


M&SIPT 

Col  Robert  McAllum 
Acquisition  Member: 
ODUSD(A&T)/SSE/DTE 


AMSWG  is  anchored  in  acquisition  community  and 
linked  to  industry  and  the  DoD  M&S  community 
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Content  of  Acquisition  M&S  Master  Plan 


Department  of  Defense 

Acquisition  Modeling  and 
Simulation  Master  Plan 


Issued  by  the 


DoD  Systems  Engineering  Forum 
April  17,  2006 

1 


•  Foreword 

*  Introduction 

•  Purpose 

•  Vision 

•  Scope 

•  Objectives  (5) 

*  Actions  (40) 

■  Action 

■  Rationale  (why  it’s  needed) 

■  Discussion  (implementation  guidance) 

■  Lead  &  supporting  organizations 

■  Products  (what  is  expected) 

■  Completion  goal  (year) 


*  Execution  Management 

http://www.aca.osd.mil/sse/as/guidance.html 


A  Decade  of  Studies  on 
M&S  Support  to  Acquisition 


1.  Final  Report  of  the  Acquisition  Task  Force  on  M&S,  1 994 

Sponsor:  DDR&E  (Dr.  Anita  Jones);  Chair:  VADM  T.  Parker,  USN  (Ret.) 

2.  Naval  Research  Advisory  Committee  Report  on  M&S,  1 994 

Sponsor:  ASN(RDA);  Chair:  Dr.  Delores  Etter 

3.  Collaborative  Virtual  Prototyping  Assessment  for  Common  Support 
Aircraft,  1995 

Sponsor:  Naval  Air  Systems  Command;  conducted  by  JHU  APL  and  NSMC 

4.  Collaborative  Virtual  Prototyping  Sector  Study,  1 996 

North  American  Technology  &  Industrial  Base  Organization;  sponsor:  NAVAIR 

5.  Application  of  M&S  to  Acquisition  of  Major  Weapon  Systems,  1996 

American  Defense  Preparedness  Association;  sponsor:  Navy  Acqn.  Reform  Exec. 

6.  Effectiveness  of  M&S  in  Weapon  System  Acquisition,  1996 

Sponsor:  DTSE&E  (Dr.  Pat  Sanders);  conducted  by  SAIC  (A.  Patenaude) 

7.  Technology  for  USN  and  USMC,  Vol.  9:  M&S,  1997 

Naval  Studies  Board,  National  Research  Council;  sponsor:  CNO 

8.  A  Road  Map  for  Simulation  Based  Acquisition,  1 998 

Joint  SBA  Task  Force  (JHU  APL  lead);  sponsor:  Acquisition  Council  of  EXCIMS  4 
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A  Decade  of  Studies  on 
M&S  Support  to  Acquisition 

9.  M&S  for  Analyzing  Advanced  Combat  Concepts,  1999 

Defense  Science  Board  Task  Force  (Co-chairs:  L.  Welch,  T.  Gold) 

10.  Advanced  Engineering  Environments,  1999 

National  Research  Council;  sponsor:  NASA 

1 1.  Survey  of  M&S  in  Acquisition,  1 999  and  2002 

Sponsor:  DOT&E/LFT&E;  conducted  by  Hicks  &  Associates  (A.  Hillegas) 

12.  Test  and  Evaluation,  1 999 

Defense  Science  Board  Task  Force  (Chair:  C.  Fields) 

13.  “SIMTECH  2007 ”  Workshop  Report,  2000 

Military  Operations  Research  Society  (Chair:  S.  Starr) 

14.  M&S  in  Manufacturing  and  Defense  Systems  Acquisition,  2002 

National  Research  Council;  sponsor:  DMSO 

15.  M&S  Support  to  the  New  DoD  Acquisition  Process,  2004 

NDIA  Systems  Engineering  Div.  M&S  Committee;  sponsor:  PD,  USD(AT&L)DS 

16.  Missile  Defense  Phase  III  M&S,  2004 

Defense  Science  Board  Task  Force  (Chair:  W.  Schneider) 


5 


Five  Objectives ,  40  Actions 


Objective  1 


Provide 
necessary 
policy  and 
guidance 


Objective  2 


Enhance  the 
technical 
framework 
for  M&S 


Objective  3 


Improve 
model  and 
simulation 
capabilities 


Objective  4 


Improve 
model  and 
simulation 
use 


1-1  M&S 

management 


1-2  Model-based 
systems 
engineering  & 
collaborative 
environments 


3  M&S  in  testing 


-4  M&S  planning 
documentatioi 


1-5  RFP  &  contract 
language 

1-6  Security 
certification 


Key 

Broader  than 
Acquisition 


2-1  Product 
development 
metamodel 

2-2  Commercial 
SE  standards 

2-3  Distributed 
simulation 
standards 

2-4  DoDAF  utility 

a)  DoDAF  2.0 
Systems 
Engineering 
Overlay 

b)  Standards  for 
depiction  & 
interchange 

2-5  Metadata 
template  for 
reusable 
resources 


3-1  Acquisition 
inputs  to  DoD 
M&S  priorities 

3-2  Best  practices 
for  model/sim 
development 

3-3  Distributed  LVC 
environments 

a)  Standards 

b)  Sim/lab/range 
compliance 

c)  Event  services 

3-4  Central  funding 
of  high-priority, 
broadly-needed 
models  &  sims 

a)  Prioritize  needs 

b)  Pilot  projects 

c)  Expansion  as 
warranted 


4-1  Help  defining 
M&S  strategy 

4-2  M&S  planning 
&  employment 
best  practices 

4-3  Foster  reuse 

a)  Business  model 

b)  Responsibilities 

c)  Resource 
discovery 

4-4  Info  availability 

a)  Scenarios 

b)  Systems 

c)  Threats 

d)  Environment 

4-5  W&A 

a)  Documentation 

b)  Risk-based 

c)  Examination 
4-6  COTS  SE  tools 

4-7  M&S  in  acqn 
metrics 


Objective  5 


Shape  the 
workforce 


5-1  Definition  of 
required  M&S 
competencies 

5-2  Harvesting  of 
commercial 
M&S  lessons 

5-3  Assemble  Body 
of  Knowledge 
for  Acqn  M&S 

5-4  M&S  education 
&  training 

a)  DAU,  DAG  & 
on-line  CLMs 

b)  Conferences, 
workshops  & 
assist  visits 

5-5  MSIAC  utility 


6 


Acquisition  M&S  Master  Plan:  Actions  1-3  &  1-4 


ACTION  1-3.  Establish  policy  and  guidance  on  appropriate 
use  of  M&S  to  plan  tests,  complement  system  live  tests,  and 
assess  joint  capabilities. 


ACTION  1-4.  Establish  policy  to  require  documented  M&S 
planning  as  part  of  the  Systems  Engineering  Plan,  T&E 
Strategy,  and  T&E  Master  Plan. 


PRODUCTS:  Revised  policy  and  guidance  in  DoDI  5000.2, 
DAG,  and  TEMP  guidance 


This  is  not  a  recommendation  to  replace  testing 
with  models  and  simulations 
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Current  Policy  Regarding  the 
use  of  Models  &  Simulations 


DoDI  5000.2;  Enclosure  5 

E5.1  The  PM,  in  concert  with  the  user  and  test  and  evaluation  communities,  shall 
coordinate  developmental  test  and  evaluation  (DT&E),  operational  test  and 
evaluation  (OT&E),  LFT&E,  family-of-systems  interoperability  testing,  information 
assurance  testing,  and  modeling  and  simulation  (M&S)  activities,  into  an  efficient 
continuum,  closely  integrated  with  requirements  definition  and  systems  design  and 
development.  The  T&E  strategy  shall  provide  information  about  risk  and  risk 
mitigation,  provide  empirical  data  to  validate  models  and  simulations,  evaluate 
technical  performance  and  system  maturity,  and  determine  whether  systems  are 
operationally  effective,  suitable,  and  survivable  against  the  threat  detailed  in  the 
System  Threat  Assessment. 

Adequate  time  and  resources  shall  be  planned  to  support  pre-test  predictions  and 
post-test  reconciliation  of  models  and  test  results,  for  all  major  test  events. 

E5.3.1  Projects  that  undergo  a  Milestone  A  decision  shall  have  a  T&E  strategy  that 
shall  primarily  address  M&S,  including  identifying  and  managing  the  associated  risk, 
and  that  shall  evaluate  system  concepts  against  mission  requirements. 


E5.4.7  Appropriate  use  of  accredited  models  and  simulation  shall  support  DT&Eg 
IOT&E,  and  LFT&E. 


Recent  Test  &  Evaluation  Policy 


Reference:  December  22,  2007  Memorandum 
Signed  by: 

Dr.  Charles  McQueary;  Director,  Operational  Test  &  Evaluation 

Mr.  John  Young,  Jr.;  Under  Secretary  of  Defense  for  Acquisition,  Technology  &  Logistics 


T&E  must  be  brought  to  bear  at  the  beginning  of  the  system  life  cycle. 

Developmental  and  operational  test  activities  shall  be  integrated  and 
seamless  throughout  the  systems  life  cycle. 

Evaluations  shall  include  a  comparison  with  current  mission  capabilities 
using  existing  data,  so  that  measurable  improvements  can  be 
determined.  If  such  evaluation  is  considered  cost  prohibitive  the 
Service  Component  shall  propose  an  alternative  evaluation  strategy. 

To  realize  the  benefits  of  modeling  and  simulation,  T&E  will  be 
conducted  in  a  continuum  of  live,  virtual,  and  constructive  system  and 
operational  environments. 
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Deputy,  Director  Developmental 
Test  &  Evaluation  Initiative 


Examine  the  formats  for  the  Test  and  Evaluation  Strategy  (TES)  and 
the  Test  &  Evaluation  Master  Plan  (TEMP) 

Either  establish  a  single  format  for  both  documents  or  make  the 
transition  from  one  to  the  other  seamless  with  a  direct  correlation 

Revise  the  format  for  the  TES/TEMP 

Provide  a  recommended  TES/TEMP  format  to  adequately  consider 
M&S 
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Deputy,  Director  Developmental 
Test  &  Evaluation  Initiative 


A  T&E  Working  Group  worked  the  initiative: 

Co-Leads:  Darlene  Mosser-Kerner,  OUSD  (AT&L) 
Tom  Carter,  DOT&E 

Participating  Organizations: 

OSD  (AT&L)  DT&E 
OSD  DOT&E 
JFCOM 
DISA 

OPNAV912 

HQ  Department  of  the  Army  (DUSA-TEO) 

COMOPTEVFOR 

HQ  Air  Force  (AQXA) 
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How  Documentation  Can  Help 


Help  a  program  manager  think  about  how  to  plan  for  and  incorporate 
M&S  into  the  T&E  process  by  identifying: 

•  how  M&S  can  contribute  to  the  T&E  process 

•  high  payoff  areas  in  which  to  invest  testing  resources 

•  the  most  cost  effective  way  of  conducting  T&E 

•  when  it  is  too  impractical  or  too  costly  to  incorporate  real  world 
assets  into  a  test  and  M&S  may  provide  insight 

•  opportunities  for  M&S  to  support  the  T&E  of  a  system  in  a  SoS 
environment 
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Original  M&S  Input  to  the  TEMP 

Format  Rejected 


Submitted  a  full  page  for  inclusion  in  the  new  TEMP  format 

Proposed  new  guidance  for  the  Defense  Acquisition 
Guidebook  (DAG)  Chapter  9 

The  full  page  submission  was  deemed  too  long  and 
the  input  was  rejected  in  total. 

Convinced  the  T&E  Working  Group  to  include  a  short 
paragraph  in  the  TEMP  Format  with  a  reference  and  link  to 
new  guidance  in  the  DAG. 
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Current  M&S  Input  for  the 
New  TEMP  Format 


2  .5  Modeling  &  Simulation  (M&S)  - 

Describe  the  key  models  and  simulations  and  their  intended  use. 

Include  the  test  objectives  to  be  addressed  using  M&S  to  include  operational  test 
objectives. 

Identify  data  needed  and  the  planned  accreditation  effort. 

(Additional  guidance  for  planning  for  the  use  of  M&S  can  be  found  at  the  DT&E 
web  page  which  DAG  Sections  4.5.7  and  DAG  Section  9.3.4  will  link  to.) 

http://www.acq.osd.mil/sse/dte/docs/M-S-Guidance-Acquisition-Workforce.pdf 
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Proposal  for  DAG  Section  9.3.4 
of  TEMP  Recommended  Format: 


•  Document  the  intended  use  of  models  &  simulations 

•  Identify  key  models  &  simulations  intended  to  support  T&E 

•  Identify  the  modeling  &  simulation  data  needed  to  support  T&E 

•  For  each  model  &  simulation  and  its  data  describe  the  planned 
accreditation  effort  based  on  the  assessment  of  the  risk  of  using  the 
model  &  simulation  results  for  decisions  being  made 

•  Describe  the  standards  (both  government  and  commercial)  with  which 
the  models  &  simulations  and  associated  data  must  comply 
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Proposal  for  DAG  Section  9.3.4 
T&E  Documentation  Planning 


Document  the  intended  use  of  models  &  simulations  by  documenting: 

•  Question(s)  to  be  answered 

•  Decisions  that  will  be  made  based  on  the  results  of  the 
models  &  simulations 

•  The  test  objectives/critical  operational  issues  the  models  &  simulations 
will  address 

•  The  requirements  for  the  use  of  the  models  &  simulations 

•  Consequences  resulting  from  erroneous  outputs  from  the  models  &  simulations 

•  Support  resources  required 
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Proposal  for  DAG  Section  9.3.4 
T&E  Documentation  Planning  (cont.) 


Identify  all  M&S  intended  to  support  T&E  (1  of  2): 

•  Live,  virtual,  and  constructive  simulations;  distributed  simulations  and 
associated  architecture;  federates  and  federations;  emulators;  prototypes; 
simulators;  and  stimulators 

•  Legacy  systems,  new  developments,  and  modified  or  enhanced  legacy 
models  &  simulations 

•  Models  &  simulations  managed  by  Federally  Funded  Research  and 
Development  Centers,  industry,  academia,  and  other  Federal  or  non- 
Federal  government  organizations 

•  Commercial-off-the-shelf  and  government-off-the-shelf  models  & 
simulations 

•  Model  &  simulation  test  resources  including  hardware-in-the  loop,  human- 
in-the-loop,  and  software-in-the-loop  simulators;  land-based,  sea-based, 
air-and  space-based  test  facilities 
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Proposal  for  DAG  Section  9.3.4 
T&E  Documentation  Planning  (cont.) 


Identify  all  M&S  intended  to  support  T&E  (2  of  2): 

•  Threat  models,  simulations,  simulators,  stimulators,  targets,  threat 
systems,  &  surrogates 

•  Synthetic  countermeasures,  test  beds,  environments,  and  battlespaces 

•  Models  &  simulations  whether  embedded  in  weapon  systems, 
implemented  as  stand-alone  systems,  or  integrated  with  other 
distributed  simulations 

•  Test  assets,  test  planning  aids,  and  post-test  analysis  tools  that 
address  other  than  real  time  characteristics 

•  Infrastructure  needed  to  conduct  a  (the)  test(s)  to  include  networks, 
integration  software,  data  collection  tools,  etc. 

•  Provide  descriptive  information  for  each  model  &  simulation  resource: 

-  Title,  acronym,  version,  date,  proponent 

-  Assumptions,  capabilities,  limitations,  risks,  and  impacts  of  the  M&S 

-  Availability  for  use  to  support  T&E 

-  Schedule  for  obtaining 
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Proposal  for  DAG  Section  9.3.4: 
T&E  Documentation  Planning  (cont.) 


Identify  the  modeling  &  simulation  data  needed  to  support  T&E: 

•  Describe  the  input  data  the  models  &  simulations  need  to  accept 

•  Describe  the  output  data  the  models  &  simulations  should  generate 

•  Describe  the  data  needed  to  verify  &  validate  the  models  &  simulations 

•  Provide  descriptive  information  for  each  data  resource: 

-  Data  title,  acronym,  version,  date 

-  Data  producer  (organization  responsible  for  establishing  the 
authority  of  the  data) 

-  Identify  when,  where,  and  how  data  was  or  will  be  collected 

-  Known  assumptions,  capabilities,  limitations,  risks,  and  impacts 

-  Availability  for  use  to  support  T&E 

-  Schedule  for  obtaining 
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Proposal  for  DAG  Section  9.3.4 
T&E  Documentation  Planning  (cont.) 


For  each  model  &  simulation  and  its  data  describe  the  planned 
accreditation  effort  based  on  the  assessment  of  the  risk  of  using  the 
model  &  simulation  results  for  decisions  being  made 

•  Explain  the  methodology  for  establishing  confidence  in  the  results  of 
models  &  simulations 

•  Document  historical  source(s)  of  verification,  validation  and 
accreditation  (VV& A)  in  accordance  with  DoDI  5000.61 

•  Provide  the  schedule  for  accrediting  prior  to  their  use  to  support  T&E 


20 


Proposal  for  DAG  Section  9.3.4 
T&E  Documentation  Planning  (cont.) 


Describe  the  standards  (both  government  and  commercial)  with  which  the 
models  &  simulations  and  associated  data  must  comply 

•  Information  technology  standards  identified  in  the  DoD  Information 
Technology  Standards  Registry  (https://disronline.disa.mil/) 

•  Standards  identified  in  the  DoD  Architecture  Framework  Technical 
Standards  Profile  (TV-1)  and  Technical  Standards  Forecast  (TV-2) 

•  Modeling  &  Simulation  Standards  and  Methodologies 
( http://assist.daps.dla.mil/) 

•  Data  standards 

•  VV&A  standards: 

-  IEEE  Std  1516.4TM  -2007,  IEEE  Recommended  Practice  for  VV&A 
of  a  Federation — An  Overlay  to  the  High  Level  Architecture  Federation 
Development  and  Execution  Process 

-  IEEE  Std  1278.  4TM  -1997(R2002),  IEEE  Recommended 
Practice  for  Distributed  Interactive  Simulation  -  VV&A 

-  MIL-STD-3022  DoD  Standard  Practice  for  Model  &  Simulation  21 
W&A  Documentation  Templates 


Summary 


Incorporating  the  use  of  modeling  &  simulation  planning  into  the  TEMP: 

•  Responds  to  new  T&E  policy  to  plan  for  using  models  & 
simulations  in  support  of  the  testing  process. 

•  Supports  the  DT&E  initiative  to  incorporate  planning  for  modeling 
&  simulation  in  the  TEMP. 

•  Addresses  recognized  needs  in  the  Acquisition  M&S  Master  Plan 

•  Provides  a  thought  process  for  a  program  manager  to  think  about 
planning  for  the  use  of  models  and  simulations  to  support  the 
testing  process 

Currently  this  is  still  work  in  progress. 
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Back  Ups 
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Current  Recommended  TEMP  Format  (DAG  9.10) 


2.  PART  ll-INTEGRATED  TEST  PROGRAM  SUMMARY 

b.  Management 

(2)  Identify  the  T&E  WIPT  structure,  to  include  the  sub-T&E  WIPTs,  such  as  a  M&S  or  Reliability, 
with  their  participating  organizations. 

3.  PART  MI-DEVELOPMENTAL  TEST  AND  EVALUATION  OUTLINE 

b.  Future  Developmental  Test  and  Evaluation. 

(3)  ....  List  all  M&S  to  be  used  to  help  evaluate  the  system's  performance,  explain  the  rationale  for 
their  credible  use  and  provide  their  source  of  verification,  validation  and  accreditation  (VV&A). 

4.  PART  IV-OPERATIONAL  TEST  AND  EVALUATION  OUTLINE 

c.  Future  Operational  Test  and  Evaluation 

(3)  ....  Whenever  M&S  are  to  be  used:  identify  the  planned  M&S:  explain  how  they  are  proposed  to 
be  used;  and  provide  the  source  and  methodology  of  the  VV&A  underlying  their  credible 
application  for  the  proposed  use. 

5.  PART  V-TEST  AND  EVALUATION  RESOURCE  SUMMARY 

a.  ...  Identify  the  following  test  resources: 

(4)  Threat  Representation:  Subject  each  representation  of  the  threat  (target,  simulator,  model, 
simulation  or  virtual  simulation)  to  validation  procedures  to  establish  and  document  a  baseline 
comparison  with  its  associated  threat  and  to  determine  the  extent  of  the  operational  and  technical 
performance  differences  between  the  two  throughout  the  life  cycle  of  the  threat  representation. 

(7)  Simulations,  Models  and  Testbeds: ...  Identify  the  M&S  to  be  used,  including  computer-driven 
simulation  models  and  hardware/software-in-the-loop  test  beds.  However,  provide  the  discussion 
of  how  these  M&S  will  be  used  in  Parts  III  and  IV.  Identify  the  resources  required  to  accredit  their 
usage.  Identify  the  M&S  Proponent,  the  V&V  Agent,  and  the  Accreditation  Agent  for  intende^Riser. 


Current  Defense  Acquisition  Guidance 
Regarding  the  use  of  Models  &  Simulations 


9.1  Introduction  to  Test  and  Evaluation  (T&E):  DT&E  supports:  the  systems  engineering  process  to  include  providing  information 
about  risk  and  risk  mitigation;  assessing  the  attainment  of  technical  performance  parameters;  providing  empirical  data  to  validate 
models  and  simulations  and  information  to  support  periodic  technical  performance  and  system  maturity  evaluations. 

The  program  manager,  in  concert  with  the  user  and  test  communities,  without  compromising  rigor,  is  required  to  integrate  modeling 
and  simulation  (M&S)  activities  with  government  and  contractor  DT&E,  OT&E,  LFT&E,  system-of-systems  interoperability  and 
performance  testing  into  an  efficient  continuum. 

9.1.5.  Integrated  T&E  Philosophy:  Live  testing  might  be  integrated  with  verified,  validated,  and  accredited  simulators  or  computer 
driven  models  and  simulations,  to  optimize  the  amount  of  live  testing  required.  Another  aspect  is  integrating  developmental  test  and 
evaluation  with  operational  test  and  evaluation  into  a  continuum  that  reduces  testing  resource  requirements  and  time,  or  conducting 
concurrent  DT  and  OT  when  objectives  and  realism  are  compatible. 

9.3.2.T&E  Working  Integrated  Product  Team:  Program  managers  should  also  consider  forming  lower  level  functional  working 
groups,  who  report  to  the  T&E  WIPT,  whose  focus  is  on  specific  areas  such  as  reliability  scoring,  M&S  development  and  VV&A, 
threat  support,  etc. 

9.3.4.  Modeling  and  Simulation  in  DT&E 

9.3.5.  System  Readiness  for  IOT&E 

9.4.1.  OT&E  Guidelines 

9.4.2.  Validation  of  Threat  Representations  (targets,  threat  simulators,  or  M&S) 

9.5.3.  Early  LFT&E 

9.5.4.  Full-Up,  System-Level  Testing  (FUSL)  and  Waiver  Process 

9.6.1. 1.  TES  Description 

9.6.2.2.  Test  and  Evaluation  Master  Plan  (TEMP)  Format 
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National  Defense  Industrial  Association 
11th  Annual  Systems  Engineering  Conference 
October  21 , 2008 


Integration  of 
MBSE  and  HSU 

v  y 


par  [constraintBlock]  StraightLineVehicleDynamics  [Parametric  Diagram] 


v.chassis.tire.  v.brake.abs.ml.  v.brake. rotor, 

friction:  dutycycle:  brakingforce: 


f  LJ  LJ  U\ 

«constraint» 

e1:BrakingForce 

f: 

«constraint» 

e2:Acceleration 

Equation  ^ 

^  [f  =  (tf*bf)*(1-tl)]  J 

F: 

^  Equation 

[F  =  m*a] 

v  J 

a: 

a: 

r  > 

^  u  > 

«constraint» 

V 

«constraint» 

e4:DistanceEquation  □ 

□  e3:VelocityEquation 

[v  =  dx/dt] 

v: 

[a  =  dv/dt] 

v _ □ _ y 

v  J 

Abe  Meilich,  Ph.D. 
Lockheed  Martin  Corporation 
abraham.w.meilich@lmco.com 


Lockheed  Martin  Copyright  2008 
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Objective  of  INCOSE  Research  activity  related  to  HSI/MBSE 
Integration 

-  What  Is  The  problem? 

-  Why  Should  You  Care? 

-  What  Is  Included  in  HSI 

-  Issues  in  Modeling  the  Human  Influence  on  System 
Design 

-  What  Is  Being  Done  Under  the  INCOSE  MBSE/HSI 
Activity? 

Summary  of  selected  HSI  modeling  and  System  Architecture 
Frameworks 

Definition  of  HSI  tasks  applied  to  SE  process 

Examples  of  Application  of  HSI  linked  to  MBSE  using  SysML 

Discussion  plans  in  2009  for  Industry,  Government,  and 
INCOSE  collaboration  in  improving  the  HSI/MBSE  interface 
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A  View  Into  the  Future 

Erosion  of  the  people/system  boundary: 

“People  will  not  just  be  users  of  the 
system  of  Ultra-Large-Scale  (ULS) 
system;  they  will  be  elements  of  the 
system,  affecting  its  overall  emergent 
behavior” 


Source:  Ultra-Large  Systems;  The  Software  Challenge  of  the  Future, 
SEI-CMU,  June  2006, 
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What  is  the  problem? 


•  Complex,  revolutionary  socio-technical  systems  pose  a  design 
problem  that  does  not  succumb  to  linear,  de-compositional 
techniques 

-  Do  we  have  SE  processes  to  deal  with  this? 

-  Predict  one  person  ?  Predict  group  behavior? 

-  Two  Air  Force  Science  Advisory  Board  (AF  SAB)  studies  have 
recognized  there  is  weakness  in  our  ability  to  better  leverage  human-to- 
human  interaction  in  the  battlespace  1 

-  The  Potomac  Institute  also  highlighted  the  lack  of  HSI  tools  to  tackle 
the  Future  of  Human  in  the  Loop 2 

-  Ring3  (2004)  argues  that  although  current  Systems  Engineering  practice 
can  be  applied  effectively  to  the  design  of  inanimate  systems,  it  faces 
significant  obstacles  in  the  design  of  human  intensive,  socio-technical 
systems. 

1  AF  SAB  2005  “System-of-Systems  Engineering  for  Air  Force  Capability  Development”,  SAB-TR-05-04 

2  Potomac  Institute  Study,  “New  Concepts  in  Human  Systems  Integration”,  March  2008 

3  Ring,  Jack  (2004).  Beyond  the  System  Operator  Paradigm;  Systems  Engineering  as  a  Sociotechnical 
System.  Conference  on  Systems  Engineering  Research,  USC/SIT/INCOSE,  April, 

2004,  Paper #120 
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What  is  the  problem? 


Our  evolving  system  of  systems  environment  demand  more 
attention  to  the  human  dimension 

-  the  elements  of  such  systems  can  together  provide  capabilities  not 
achievable  in  isolation  -  leveraging  the  power  of  networking 

-  definitions  of  the  boundaries  of  these  elements  create  dependencies 
and  interaction  activities  -  emergent  behavior  (both  bad  and  good) 

-  the  mission  performance  of  such  systems  is  greatly  improved  through 
attention  to  the  resulting  human  communication  and  coordination 
efforts  -  often  overlooked 

Why  are  the  products  of  cognitive  engineering  ignored  in  the 
systems  development  process? 

-  It  is  not  because  the  challenges  of  Human-System  Integration  (HSI)  are 
unrecognized  but  because  the  products  of  cognitive  engineering  do  not 
resonate  with  the  design  community  at  large1 


1  Lintern,  Gavan,  “Human  Performance  Modeling  for  Enterprise  Transformation,  Proceedings  of  the  16th 
Annual  International  Symposium  of  the  International  Council  on  Systems  Engineering,  2006 
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Another  recommendation 


•  Use  of  scenario  based  analysis  advocated’ 


Recommendation:  Adapt  existing  or  develop  new  methods  and  tools 
that  facilitate  capture  and  traceability  of  HSI  design  objectives,  design 
rationale,  and  constraints  across  design  phases.  Specifically: 


Adapt  existing  and  develop  new  methods  tor  generating  scenarios 
that  reflect  the  range  of  complexities  uncovered  by  context  of  use 
analyses.  This  corpus  of  scenarios  can  be  used  to  support  develop¬ 
ment  and  evaluation  of  designs,  procedures,  and  training,  including 
human  reliability  and  safety  analyses.  They  could  also  be  used  to 
exercise  models  anti  simulations  as  part  nf  the  system  development 
process.  The  goal  would,  be  to  ensure  tnat  the  systems  have  been 
explicitly  designed  and  tested  to  support  performance  across  a 
comprehensive  range  of  representative  situations,  as  identified  by 
context  of  use  analyses.  Context  of  use  scenarios  are  also  essential 
to  the  meaningful  definition  of  such  key  performance  parameters 
as  response  time,  reliability,  and  accuracy. 


Human-System  Integration  in  the  System  Development  Process:  A  New  Look”, 

Committee  on  Human-System  Design  Support  for  Changing  Technology, 

Richard  W.  Pew  and  Anne  S.  Mavor,  Editors,  Committee  on  Human  Factors,  National  Research  Council, 
The  National  Academy  Press,  p  306,  2007. 


One  of  several 
recommendations 
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Human  Systems  Integration:  Mandate 


AFSAB  Report,  SAB-TR-04-04 


Study  Overview 


At  R  FORCE 


Increased  demands  on  human  operators 

•  Volume  and  complexity  of  information 

•  Changing  job  demands 

•  Manpower  constraints 


Impact  on  system  effectiveness 


Study 

Assessment: 

Shortfalls  in  HSI  Practices 


■  Accuracy  and  timeliness  of  decisions 

■  Operational  safety 

•  Acquisition  cost  and  schedule 

•  Total  system  life  cycle  cost 


Lack  of  organizational  focus  &  advocacy 
No  definitive  AF  policy/program  guidance 
Lack  of  measurable  requirements 
Resources  below  critical  mass 
Inconsistent  planning  and  execution 
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Potential  Solution? 


Potential  Solution:  Leverage  and  adapt  new  methods 
of  SE  modeling  (MBSE)  techniques  to  help  the 
construction  of  a  bridge  between  cognitive  engineers, 
as  well  as  all  HSI  domains,  and  systems  engineers 
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What  is  included  in  HSI? 


/ 


Manpower 

Staff  count  and  composition;  total  cost. 

Personnel 

Required  and  available  personnel  skills  and  aptitudes;  physical  abilities;  security  clearances;  retention 
or  attrition  rates;  total  cost. 

Training 

Types  of  training  and  lengths  of  training;  recurrent  training  requirements;  impact  of  training  on 
readiness;  total  cost  of  training. 

Human  Factors 
Engineering 
(HFE) 

Required  human  capabilities;  usability  of  proposed  system;  task  performance  times;  accuracy  (error 
rates)  and  efficiency  (number  of  tasks  performed  in  a  given  time  period);  cognitive  and  physical 
workloads;  stress;  organizational  impact;  effectiveness  of  communications. 

Safety 

Potential  for  errors  that  cause  injury;  potential  for  loss  of  use  of  system;  potential  for  loss  of  personnel; 
cost  of  implementing  reasonable  safety  precautions. 

Occupational  Health 

Health  hazards;  severity  and  risks  associated  with  hazards;  total  cost  to  minimize  hazards  or  their 
consequences. 

Survivability 

Probability  of  being  detected,  attacked,  or  mistaken  for  enemy;  ability  to  minimize  injury;  ability  to 
minimize  physical  or  mental  fatigue;  total  cost  of  reducing  risks. 

Verification  and 
Validation 

Human  system  requirements  met;  functionality  exists  to  accomplish  the  tasks  or  functions  required; 
results  compared  to  other  sources  to  confirm  accuracy  within  acceptable  tolerances. 

Note:  most  recently  more  areas  have  been  proposed  under  the  HSI  umbrella  »»> 
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HSI  needs  to  communicate  and 
inform  SE 


■Differences  in  terminology1- 


Term 

SE  interpretation 

HSI  interpretation 

Task 

A  high  level  description  of  what  an 
Enterprise  needs  to  achieve. 

A  duty  that  individuals  carry  out  as  part 
of  their  job. 

Activity 

A  high-level  description  of  what  needs 
to  be  achieved,  before  individual 
resources  are  specified. 

A  low-level  description  of  what  individual 
people  may  do  as  part  of  their  tasks. 

Function 

A  specific  description  of  what  individual 
resources  are  designed  or  designated  to 
do  (e.g.  human,  machine,  animal). 

A  generic  description  of  what  needs  to  be 
done  at  a  high  level  of  task  descriptions  - 
often  resource-independent. 

Role 

Something  to  be  done  that  is  defined 
independently  of  whether  a  human  or  a 
machine  will  carry  it  out  -  since  these 
allocations  may  change. 

Something  to  be  done  by  people  (mostly 
one)  who  take  responsibility  for  the 
outcomes.  This  is  closely  related  to  job 
definitions. 

Bruseberg  A  (In  press)  “Human  Views  for  MODAF  as  a  Bridge  between  Human  Factors  Integration 
and  Systems  Engineering”.  Cognitive  Engineering  and  Decision  Making  Journal. 

(Special  Section  on:  Integrating  Cognitive  Engineering  in  the  Systems  Engineering  Process: 

Opportunities,  Challenges  and  Emerging  Approaches.)  Publisher:  Human  Factors  and  Ergonomics  Society,  2008  10 
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Integration  of  Hardware, 
Software,  &  Human  Life  Cycles 

A 


Design  Requirements 

(criteria)* 


Design  for: 

•  Performance 

•  Cost-system  effectiveness 

•  Reliability 

•  Maintainability 

•  Political,  Social,  &  Tech 
Feasibility 


Human  Factors 
Safety 
Environment 
Occupational  Health 
Manpower 
Personnel 
Training 
Survivability 
Habitability 


•  Vulnerability 

•  Supportability 

•  Producibility 

•  Reconfigurability 

•  Affordability 

•  Disposability 

•  Flexibility  (growth) 

•  applicable  to  all  levels  in  the 
system  structure  and  tailored 
to  specific  program  needs 


J 


Design  accomplished 
through: 

•  Requirements  analysis 
•Quality  function 

deployment 

•  Feasibility  analysis 

•  Operational  requirements 
&  maintenance  concept 

•  Functional  analysis 

•  Design  trade-off  studies 

•  Simulation  &  modeling 

•  Requirements  allocation 

•  Reliability  &  maintainability 
analyses 

•  Human  system  integration 

•  Supportability  analysis 
•Test  and  evaluation 

•  Risk  analysis 

•  Other  supporting  analyses 


Requirements  Analysis 

■;« 

Functional  analysis 
(systems  level) 

1 

X 


Functional 

Group 

hardware 


Equipment 

~~r~ 


Equipment 
&  Accessories 


Hardware 

Structure 


Component 
Integration  & 
prototypes 


X 


Equipment 

Testing 

— 1= 


Functional 

Group 

software 


Functional 

Group 

human 


Computer 

Software 

units 

~~r~ 


Human 

Activities/duties 


Software 

Configuration 


X 


X 


Human 

Tasks/Subtasks 


Software 

Structure 


X 


MP 

Requirements 


Software 

Component 

integration 


Software 

Testing 

=P= 


Personnel 
Development  & 
Training 


Personnel 

Testing 

=1 - 


Day-to-day  design 
Integration  activities 


Evaluation 
(system  integration 
And  testing) 

t 

Source:  Modified  graphic  from  Blanchard  &  Fabrycky,  Systems  Engineering  and  Analysis,  2006,  pp.  106 
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Human-Centered  Tasks  in  System  Life  Cycle 


Conceptual 

Preliminary 

Detail  Design  and 

Production  and/or 

System  Operation 

Design 

System  Design 

Development 

Construction 

and  Support 

System 

Requirements 

Human  Systems 
Requirements 
Human  Systems 
Plan 


•Functional  Allocation 
•Operator  Task  Analysis 
•Operational  Sequence 
Diagrams 

•Human  Error  Analysis 
•Operator  Safety/Hazard 
analysis 


•Operational  Requirements 
•Maintenance  Concept 
•Tech  Perform  Measures 
•Functional  Analysis  &  Allocation 


Design  Review  and  Integration 
Human  Factors  and  Safety  Analysis 
Personnel  and  Training  Information 


’Design  Participation 
‘Human-System  Interface 


•Personnel  Training  Analysis 
yj  ‘Training  Equip/Software  Design 


Personnel  Test  and  Evaluation 


Data  Collection,  Analysis,  and  Corrective  Action 
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Issues  in  Modeling  the  Human  Influence  on  System 
Design 

•  HSI  modeling  has  remained  in  the  HSI  domains 

-  No  way  of  linking  HSI  models  to  SE  models  due  to  domain  languages  and  lack 
of  relevant  taxonomy  linkage  to  SE  needs 

•  It  is  challenging  to  link  the  soft  behaviors  of  the  human  to  the 
predictable  behaviors  of  machines 

-  Human  performance  modeling  issue  -  cognitive  capability  and  capacity  can 

change  with  stress,  fatigue  and  experience.  Sometimes  the  direction  of 
change  can  be  unexpected  (e.g.,  team  performance  under  high  workload  can 
exhibit  emergent  behavior)  ^ - 

•  There  is  lack  of  awareness  of  what  attributes  ofnthnqn  behavior  can  be 
linked  to  system  effectiveness  as  it  relates  to  overall  mission 
effectiveness;  thus  limiting  the  ability  of  an  SE  to  perfornrtade  studies 

•  Note  this  issue  as  discussed  by  the  AF  SAB  *:  \ 

-  “Whenever  the  Air  Force  generates  a  svstem-of-svstems,  interact  n  among 
the  systems  often  includes  human-to-human  interactions.  If  the  mpchine-to- 
machine  aspect  of  SoS  is  weak,  then  it  falls  upon  the  humans  to  /chieve  the 
interaction.  This  can,  and  often  does,  create  a  very  challenging yhvironment 
for  the  human;  sometimes  leading  to  missed  opportunities  orsrerious 
mistakes.  The  lack  of  sound  Human  System  Interface  designsrcan  exacerbate 
this.  Coordinated  situation  awareness  is  difficult  to  manaaaflf  the  individual 
systems  miss  or  convey  contusing  or  conflicting  information  to  their 
operators.” 

*  AF  SAB  2005  “System-of-Systems  Engineering  for  Air  Force  Capability  Development”,  SAB-TR-05-04 
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What  is  being  Done  Under  the 
INCOSE/HSI  Tasking? 


•  Evaluate  how  present  MBSE  artifacts  can  be 
related  to  SE  artifacts  from  various  HSI  modeling 
approaches  (including  cognitive  model 
applications)  in  practice  today 

-  Leverage  HSI  WG  at  INCOSE  and  other 
industry  forums 

-  Link  to  systems  models  in  SysML 

-  Link  to  dynamic  models  from  system 
dynamics  theory 

-  Link  to  experimentation  techniques 

-  Link  to  executable  cognitive  architecture 
representations 
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Initial  findings 


•  Many  tools  and  computational  engines  used  to 
perform  HSI  analysis 

-  In  process  of  negotiating  prototypes  of 
linking  ( automatically  or  semi-automatically) 
HSI  data  with  SE  data  in  a  MBSE  environment 

-  IMPRINT™  to  be  used  in  conjunction  with 
SysML  for  first  prototype.  Others  are  being 
investigated  for  prototypes 

•  LMC  developing  a  HSI/SE  methodology  that  can 
leverage  MBSE  modeling  techniques  to  perform 
more  “human  centric”  SE 

-  Results  to  be  reported  at  Winter  2009  INCOSE 
Workshop 
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What  modeling  techniques  are  out  there 
for  integrating  HSI  with  SE 


Initial  Research: 

•  IMPRINT  (Dynamic  modeling  of  human 
performance  characteristics  in  a  system  - 
US  Army  tool) 

•  SysML  (common  standards  based  SE 
language  for  modeling) 

•  Architecture  Frameworks  (Human  Views) 

•  SOA  Services  and  Standards 
(BPEL4People) 
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MBSD  Encompasses  Multiple  Modeling  Domains  ^ 


MBSD  Integration 


HSI  Analysis  Models 


U(s) 


G(s)  -) 


Hardware  Models 


S  SET  Q 

— 

> 

/ 

R  CLR  Q 

he 

Software  Models 


A 

nr 

/ 

i - 1 
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System  Engineering  Technical  Life  Cycle  Processes/ 

•ISO/IEC  JTC1/SC7  15288,  Systems  and  Software  Engineering  -  System  Life  Cycle  Processes 


Stakeholder 

Requirements 

Definition 


Disposal 


Requirements 

Analysis 


liiiilii! 

aiiipii 

'■■■■I 


Architectural 

Design 


Implementation 


Verification 


Maintenance 

v5  .  1  i;li 
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X  :  Wr  \ 

Operation 

Validation 

Transition 
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HSI:  A  Cornerstone  of  Human  Performance 


Human  Performance 


Human 

Capabilities/ 


Human 


IjA'fijf"  r 


Human 

Fitness 


Air  force 
example 


Human- 

Machine 

Knowledge, 
Skills  and 

Crew 

Work 

Airmen  are^ 
qualified,  rested, 

CJUjbu|gn 

Human  Systems  Integration 


Environment 

Personnel  Training  Manpower  Safety  &  Habitability  Survivability 


Hardman,  N.,  Colombi,  Jacques,  D.  and  Hill,  R.,  “What  System  Engineers  Need  to  Know  About  Human-Computer 
Interaction”  Lockheed  Martin  Copyright  2008 
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Inputs  and  Outputs  to  SE/HSI  Models 


SE 

Process 


Concept  Parameters:  (Type  of  System.  Customer  Goals.  Target  Roles.  Constraints) _ 

lRe9ilirements)  HSI  Parameters:  (Expert  Knowledge.  Task  Steps.  Cognitive  Processes.  Work-arounds) 

Requirements  Parameters  (CONOPS,  System  Requirements.  Operational  Requirements) , 
HSI  Parameters  (Operational  CONOPS,  Human  Performance  (HP)  Reus.  HP  Metrics) 


Design  Parameters  (Architecture  Design,  System  Design) 


Termination  Parameters  (Disposal  artifacts) 


HSI 


HSI  Functional  Parameters  (Interaction  Paradigm.  Function  Allocation.  Workloads 

Development  Parameters  (System  Components.  Low-fidelitv  Prototypes  ) _ 

3ie i  entation)  |-|SI  Design  Parameters  (Changes  Based  on  Usability  &  User  Interface  Standards) 

Development  Parameters  (Hiaher-fidelitv  Prototypes) _ 

HSI  Support  Parameters  (Training  Materials.  User  Manuals) _ 

Testing  Parameters  (Test  Plan,  System  Metrics) _ 

j)  HSI  Testing  Parameters  (Changes  Based  on  Usability  &  User  Interface  Standards) 

Transition  Parameters  (System  ) _ 

)n  )  HSI  Transition  Parameters  (Times  &  Probabilities  of  Competing  Sequences  of  Task: 

Testing  Parameters  (System  Performance  in  Intended  Environment ) _ 

Testing  Parameters  (HSI  MOE  and  MQPt _ 

Performance  Parameters  (System  Performance ) _ 

HSI  Performance  Parameters  (Training  vs  Pprfnrmannp  ) _ 

Maintenance  Parameters  (Personnel  &  Training  Costs) _ 

(Maintenance)  HSI  Maintenance  Parameters  (Personnel  Expertise  &  Training  Modifications^ 


Heuristic 


Tuman-ir 
the-Lc 


HSI  Termination  Parameters  (Lessons  Learned.  Replacement  Guidelines  for  Us* 


HSI 

Analysis 

Methods 


T 
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Ongoing  UML  -  IMPRINT  Pilot  Study  Project  ^ 


Source:  Presentation:  “Enhancing  System  Design  by  Modeling  IMPRINT  Task  Workload  Analysis  Results  in  the 
Unified  Modeling  Language”,  Diane  Mitchell,  Operations  Analysis  Team  Leader,  Integration  Methods  Branch,  US 
Army  Research  Laboratory,  ,  2008. 
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Architecture  Framework  Products  Supporting  HSI/MBSE 


Fiaure  4:  Human  Views  in  Context, 


Source:  “Human  Factors  Integration  for  MODAF:  Needs  and  Solution  Approaches”, 
A.  Bruseberg  &  G.  Lintern,  INCOSE  Annual  Symposium  2007 


Also  see:  Bruseberg  A  (In  press)  “Human  Views  for  MODAF  as  a  Bridge  between  Human  Factors  Integration 
and  Systems  Engineering”.  Cognitive  Engineering  and  Decision  Making  Journal. 

(Special  Section  on:  Integrating  Cognitive  Engineering  in  the  Systems  Engineering  Process:  23 

Opportunities,  Challenges  and  Emerging  Approaches.)  Publisher:  Human  Factors  and  Ergonomics  Society,  2008 
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Architecture  Framework  Products  Supporting  HSI/MBSE 

( Another  view  of  MODAF/HVt 


HV-A: 

Personnel 

Availability 


HV-C:  Human 
Interaction 
\  Structure 


HV-B:  Quality 
Objectives  and 
Metrics 


Organisation 


HV-G:  Dynamic 
Drivers  of  Human 
Behaviour 


OV-5 


Roles  and 
Competencies 


HV-E:  Human 
Functions  and 


The  Human  View  Handbook  for  MODAF”,  Systems  Engineering  &  Assessment,  Ltd, 
Produced  on  behalf  of  the  MoD  HFI DTC,  ©  Crown  Copyright,  Bristol,  UK,  15  July  2008 

http://www.hfidtc.com/MoDAF/HV  Handbook  First  lssue.pdf 


Figure  8:  An  overview  of  HVs  for  MODAF  in  relation  to  MODAF  Views  at  different  levels 
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SOA  Services  And  Human-in-the- 
Loop 


Process 

Improvement 

OMG  -  Business  Process  Maturity  Model  (BPMM) 

Process  Modeling 

OMG  -  Business  Process  Modeling  Notation  (BPMN) 

OMG  -  Business  Process  Definition  Meta-Model  (BPDM) 

WFMC  -XML  Process  Definition  Language  (XPDL) 

Task  Management 

WS-HumanTask 

Process  Execution 

OASIS  -  Business  Process  Execution  Language  WS  BPEL  2.0 

WS-BPEL  Extension  for  People 

(http://www- 

128.ibm.com/developerworks/webservices/library/specification/ws- 

bpel4people) 

\ 

Orchestrate  people,  systems,  content,  and  business  rules  into  streamlined 
end-to-end  processes  that  are  accessible  to  process  participants  through 

engaging  user  interfaces,  online  or  offline. 

v  J 
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BPEL4People  features  _ 

Features  addressed  by  WSHumanTask 

Human  Task  Behavior 

-  Normal  Processing  of  a  Human  Task 

-  Releasing  a  Human  Task 

-  Delegating  or  Forwarding  a  Human  Task 

-  Suspending  and  Resuming  a  Human  Task 

-  Skipping  a  Human  Task 

-  Termination  of  a  Human  Task 

-  Error  Handling  for  Human  Task 

Other  considerations: 

•Scope  of  users  (i.e.,  operators,  management,  stakeholders,  etc.) 

•User  Interfaces  to  Applications 
•Portability  and  Interoperability  Considerations 

-  The  portability  and  interoperability  aspects  Features  addressed  by  WSHumanTask: 

•  Portability  -  The  ability  to  take  human  tasks  and  notifications  created  in  one  vendor's  environment  and 
use  them  in  another  vendor's  environment. 

•  Interoperability  -  The  capability  for  multiple  components  (task  infrastructure,  task  list  clients  and 
applications  or  processes  with  human  interactions)  to  interact  using  well-defined  messages  and 
protocols.  This  enables  combining  components  from  different  vendors  allowing  seamless  execution. 
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How  can  MBSE  and  SysML  help? 


•  Various  efforts  are  underway  to  leverage 
SysML  as  part  of  Systems  Engineering 
analyses 

-  SysML  is  a  System  Engineering 
Modeling  Language  -  a  superset  of 
UML 
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Example  Integration  of  HSI  and  MBSE 


Activity 

Diagram 


Behavior 

• 

• 

l 

Requirement  | 

Structure 

r  7 ram 

l 

l 

Diagram  1 

Diagram 

5 


Sequence 

State  Machine 

Use  Cas  ^ 

1  Block  Definition 

Internal  Block 

i 

ackage  Diagram 

Diagram 

Diagram 

Diagram 

1  Diagram 

Diagram 

Same  as  UML  2 
Modified  from  UML  2 
New  diagram  type 


Parametric 

Diagram 
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IMPRINT™  Example  -  0V-6b  Operational  State  Transition 
Diagram 


k 


29_0  Start  ->*4  29_1  TC  Initiates  Communication 


29_5 TC Transmits  Communication  <$>  >4  29_999End  <X 


Weapons  Control 


Weapons  Control 


-X 


Idle 


Ke 


whjn 


WCO  Arrives/ 


V 


Processing  Qders 


Entry/Prepare  For  Attack 


Ops  Control  ~j~ 


Order  times  "out )/ 

rSlrki 
L1ELCC«»1V 


[Ready  t 


R#qu 
T  angelln 


Prices: 

Tear 


Plainer 


-> 


got 


Esecuti 


Inle  co 


30_1  Acquire  Target 


Live 


V 


X  Waiting  fa  Target 


[within  valid  target  are^ 
]/Detect  Target 


Acquiring  Target 


Entry/ Acquire  Target 


/Target  Acquired 


Atacking  Target 


Entry/Attack  Target 


Target  Engagement  Ended/ 
Acquire  Attack  Intelligence 


-■e 


Data  objects: 

Operational  states 
Events 

Operational  state  transitions 
Usage: 

Operational  analysis 


i/ers/on  or  Uv-ou 


30_n+l  Engage  Target 


Description: 

Alternative  Views: 

Graphical  method  of  describing  how  an  operational 

UML  Statechart  Diagram 

node  or  activity  responds  to  various  events  by 

changing  its  state 

Source:  I MPRI NT/Artisan  Software  ChartS)ck2008  artin  Copyright  2008 


User  Characteristics 


«User» 

{Age  =  13-100 
Elderly  may  have  limitations} 

{Computer  Experience  =  Minimal} 
{Disability  =  Upper  body  movement 
Minimal  Sight  required 
May  need  large  buttons 
Hearing  for  alarms  -  Alternative 
Flashing  Lights?} 

{Frequency  =  Undefined} 

{Language  =  English/May  need  internationalisation} 
{Motivation  =  Keep  House  and  belongings  safe 
Save  time,  save  money} 

{Sex  =  M/F} 

{Task  Consistency} 


«User» 

{Age  =  18-65} 

{Computer  Experience  =  Advanced,  Detailed  H/W  Knowledge} 
{Disability  =  Normal  Sight,  Hearing  and  Mobility} 
{Frequency  =  Regular} 

{Language  =  English} 

{Motivation  =  Maintain  System  in  Good  Working  Order 
Minimise  False  Alarms 
Minimise  System  Faults 
Maintain  Professional  Company  Image} 

{Sex  =  M/F} 

System 

Maintainer 


Basic  User 


{Age  =  13-100} 

{Computer  Experience  =  Minimal} 
{Disability  =  Upper  body  movement 
Minimal  Sight  required 
May  need  large  buttons 
Hearing  for  alarms  -  Alternative 
Flashing  Lights?} 

{Frequency  =  Occasional} 

{Language  =  English/May  Need  internationalisation} 
{Motivation  =  Keep  House  and  belongings  safe 
Save  Time,  Save  Money} 

{Sex  =  M/F} 

{Task  Consistency} 

Occasional 

User 


{Computer  Experience  =  Understanding  of 
Menu  Driven  Systems} 

{Disability  =  Upper  Body  Movement 
Normal  Sight  Required 
Hearing  For  Alarms} 

{Frequency  =  Regular} 

{Language  =  Native  English} 
{Motivation  =  Keep  House  and  belongings  safe 
Save  Time,  Save  Money 
Ensure  System  Works  Correctly} 

{Sex  =  M/F} 

Regular  User 


«User» 

{Age  =  18-70} 

{Computer  Experience  =  Advanced} 
{Disability  =  Upper  Body  Movement 
Very  Good  Sight  for  Small  Components 
Hearing  For  Alarms 

Mobility  through  house  to  check  components 
May  need  to  reach  high  places} 
{Frequency  =  Regular} 

{Language  =  English} 

{Motivation  =  Keep  House  And  belongings  Safe 
Save  Time,  Save  Money 
Ensure  System  is  in  Good  Working  Order 
Prevent  Future  Faults} 

{Sex  =  M/F} 

Advanced 
User 


Source:  I MPRI NT/Artisan  Software  ChartSx±2008  artin  Copyright  2008 
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Task  Characteristics 


Basic  User 


Occasional 

User 


Regular  User x 


«Task» 

{Task  Variance  =  Minimal} 
{Frequency  =  Regular} 

{Reqrd  Skills  =  Task  Familiarity} 
{Safety  Issues  =  None} 

{Reqrd  Users  =  Normally  Single  User} 
{Task  Consistency} 

Schedule 
Appliance 


Advanced 

User 


«Task» 

{Task  Variance  =  Medium} 
{Frequency  =  Occasional} 

{Reqrd  Skills  =  Appliance  Familiarity 
Ul  Expertise 
Some  Fl/W  Knowledge} 

{Env  Impact} 

{Safety  Issues  =  Dependent  on  Appliance} 
{Reqrd  Users  =  Normally  Single 
Can  involve  others  for  Test} 
Manage 
^Appliance^ 


«Task» 

{Task  Variance  =  Minimal} 
{Frequency  =  Regular} 
{Reqrd  Skills  =  Basic  Computer} 
{Safety  Issues  =  None} 
{Reqrd  Users  =  Single} 
Access 
System 


«Task» 

{Task  Variance  =  Minimal} 
{Frequency  =  Regular} 
{Reqrd  Skills  =  Minimal  S/W} 
{Safety  Issues  =  None} 
{Reqrd  Users  =  Single} 
Logoff  System 


«Task» 

{Task  Variance  =  Minimal} 
{Frequency  =  Regular} 

{Reqrd  Skills  =  S/W  Ul} 

{Time  Critical} 

{Safety  Issues  =  May  require  verification  or 
proximity  test} 

{Reqrd  Users  =  Single} 

Upload 
Schedule 


«Task» 

{Task  Variance  =  Maximum} 
{Frequency  =  Occasional} 
{Reqrd  Skills  =  H/W,  S/W  Expertise} 
{Env  Impact} 

{Time  Critical} 

{Safety  Issues  =  FISE  Electrical,  etc} 
{Reqrd  Users  =  Normally  Single 
Safety  May  require  2} 
Upgrade 
Hardware 


«Task» 

1 1  ask  Variance  =  Maximum} 
{Frequency  =  Occasional} 
{Reqrd  Skills  =  S/W  Expertise} 
{Env  Impact} 

{Safety  Issues  =  Certification  Required 
for  some  devices} 

{Reqrd  Users  =  Normally  Single  User} 
Update 
Software 


«Task» 

{Task  Variance  =  Minimal} 
{Frequency  =  Occasional} 
{Reqrd  Skills  =  S/W,  Some  H/W} 
{Safety  Issues  =  None} 
{Reqrd  Users  =  Single} 
Basic  System 
Maintenance 


«Task» 

{Task  Variance  =  Maximum} 
{Frequency  =  Occasional} 
{Reqrd  Skills  =  H/W,  S/W  Expertise} 
{Env  Impact} 

{Time  Critical} 

{Safety  Issues  =  HSE  Electrical} 
{Reqrd  Users  =  Single 
Sometimes  double  for  safety} 
Advanced  System 
Tlaintenan 


System 

Maintainer 


Source:  I MPRI NT/Artisan  Software  ChartSx±£008  artin  Copyright  2008 


Parametrics 

•  Used  to  express  constraints  (equations)  between  value  properties 

-  Provides  support  for  engineering  analysis 
(e.g.,  performance,  reliability) 

-  Facilitates  identification  of  critical  performance  properties 

•  Constraint  block  captures  equations 

-  Expression  language  can  be  formal  (e.g.,  MathML,  OCL)  or 
informal 

-  Computational  engine  is  defined  by  applicable  analysis  tool  and 
not  by  SysML 

•  Parametric  diagram  represents  the  usage  of  the  constraints  in  an 
analysis  context 

-  Binding  of  constraint  usage  to  value  properties  of  blocks  (e.g., 
vehicle  mass  bound  to  F=  m  x  a) 


Parametrics  Enable  Integration  of  Engineering  Analysis  with 

Design  Models 


Lockheed  Martin  Copyright  2008 


Vehicle  Dynamics  Analysis  (example) 


Lockheed  Martin  Copyright  2008 


Future  Plans  for  INCOSE  HSI/MBSE 
Collaboration  in  2009 


*  Develop  an  initial  mapping  between  the  artifacts 
produced  in  SE  Process  to  HSI  Process/analysis 

*  Map  HSI  artifacts  into  static  structural  modeling 
framework  including  interdependency  across 
systems. 

*  Comprehensive  Example  Architecture:  Using 
MBSE  approach  with  an  exemplar  architecture 
using  the  outcomes  of  2008  effort 

*  Develop  example  integration  of  HSI  tool  to  MBSE 
environment  (  e.g.,  using  SysML) 

*  Work  with  HSI/SE  community  to  help  peer  review 
approaches  developed  under  our  INCOSE 
activity 
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Systems  Engineering  Re¬ 
vitalization  at  the  Defense 
Contract  Management  Agency 


Presented  By: 

Mr.  Lawrence  Cianciolo 


October  20-23,  2008 


DCMA  AGENDA 


Defense  Contract  Management  Agency 


Charter 

Feedback 

DCMA  Systems  Engineering  Value  to  the  DoD  Acquisition 
Enterprise 

DCMA  Systems  Engineering  Functions  and  Influence  Areas 
DCMA  Systems  Engineering  Core  Processes 
Recommended  Path  Forward 

*  Baseline  Skills  Assessment 

*  Competency  Training 

*  Develop  Policy/Tools/Guidance 

*  Recommended  Training  Track/Curriculum 

*  SE  Standard  Surveillance  Operating  Guide  (SSOG)  Outline 
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DCMA  Charter 


Defense  Contract  Management  Agency 


Intent  is  to  define  expectations  and  prioritize 
processes,  functions,  and  efforts  of  DCMA  engineers 
in  providing  the  best  guidance,  support,  and  life-cycle 
balanced  system  solutions  that  satisfy  customer 
needs,  goals,  objectives,  requirements,  and  specific 
outcomes  in  DoD  weapon  systems  acquisition 
management 


Defining  the  Future  of  DCMA  Systems  Engineering! 


DCMA  Feedback 


Defense  Contract  Management  Agency 


Feedback  on  our  recommendations  -  provided  by  OSD  (AT&L),  PEO, 
DCMA  Division  Director,  CMO  Commander,  and  CMO  Engineers: 

"...a  sound  approach  with  a  great  explanation ...”  -  Dr.  Don  Gelosh, 
Senior  Systems  Engineer ,  OSD  (AT&L)  SSE/ ED 

“...you  have  a  good  handle  on  this...”  -  Col  Rich  Hoeferkamp, 
Acting  Deputy  Director ,  OSD  (AT&L)  SSE/ ED 

“...you  are  on  the  right  track...”  -  Alex  Levi ,  PEO  Staff  Engineer, 
Space  and  Missile  Systems  Center,  Los  Angeles  AFB 

“...this  initiative  is  much  needed...”  -  Col  Warren  Anderson,  DCMA 
Dayton  Commander  -  OSD  (AT&L)  SE  Instructor 

“I like  the  Engineering  Core  Processes  listed...”  -  Gregory  Lehn, 
P.E.,  DCMA  NASA  Product  Operations 
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DCMA 


Defense  Contract  Management  Agency 


DCMA  Systems  Engineering  Value  to  the  DoD 

Acquisition  Enterprise 


*  Primary  Result  of  OSD  (AT&L)  Study 

•  A  lack  of  Systems  Engineering  process  capability 
and  process  compliance  were  primary  contributors 
to  poor  program  performance 

*  Revitalizing  DCMA  Systems  Engineering  efforts 
would  help  to  improve  program  performance 

*  Aligns  with  OSD  (AT&L)  Mission  to  Revitalize 
Systems  Engineering  Throughout  the  DoD 


DCMA  SYSTEMS  ENGINEERING  FUNCTIONS  AND 

INFLUENCE  AREAS 


Defense  Contract  Management  Agency 


DCMA  Systems  Engineers 

*  Ensure  that  the  contractor  has  effective  processes 

*  Ensure  that  the  contractor  delivers  products  that 
meet  requirements  and  are  delivered  on  schedule 
and  within  cost 

*  Track  cost,  schedule  and  technical  performance, 
perform  risk  analysis,  perform  predictive  analysis 
of  program  impacts,  and  recommend 
improvements  to  contractor  performance 

*  Influence  the  contractor  to  improve  performance 

*  Provide  needed  recommendations  to  the  PMO 


DCMA 


Defense  Contract  Management  Agency 


DCMA  SYSTEMS  ENGINEERING  FUNCTIONS 

AND  INFLUENCE  AREAS 


*  Ensure  Products  Meeting  Customer  Requirements 
in  a  Timely  Manner  (Satisfied  Customer) 

*  Support  Major  Program  Performance  Commitments 
(PCs) 

*  Perform  Mandated  DCMA  Systems  Engineering 
Activities  in  Support  of  Certain  MOAs 


These  Functions  are  Implemented  via  the 
DCMA  Systems  Engineering  Core  Processes 


DCMA  Engineering  Core  Processes 


Defense  Contract  Management  Agency 


DCMA 


Defense  Contract  Management  Agency 


Recommended  Path  Forward 


Establish  DCMA  HQ  Sys  Eng  Competency  Team 

*  Baseline  Skills  Assessment 

*  Assess  Core  Processes,  roles  and  responsibilities  - 
continuously  review  for  modifications 

•  Baseline  core  competency  skills  needed  by 
commodity 

•  Identify  skills  needed  to  implement  new  technology  in 
future  programs 

•  Identify  skills  needed  to  sustain  legacy  systems 

•  Align  with  AT&L  Competency  Assessment  Efforts 


Engineering  Disciplines  are  Unique 


DCMA  Recommended  Path  Forward 


Defense  Contract  Management  Agency 


Establish  DCMA  HQ  Sys  Eng  Competency  Team 

*  Develop  Competency  Training  Program 

*  Consolidate  and  prioritize  Division  training  inputs 

*  Secure  Systems  Engineering  training  funding 

*  Execute  Systems  Engineering  training 

*  Define  training  standards  and  timelines 

*  Measure  Agency  training  success  (Metrics) 

*  Develop  measures  of  success  to  achieve  core 
competencies 

*  Integrate  results  with  the  following  recommended 
training  path/curriculum: 
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DCMA 


Defense  Contract  Management  Agency 


Recommended  Training 
Path/Curriculum 


Training  Type 

Course(s) 

Education 

Appropriate  ABET  Accredited 
Degree 

DAWIA 

Level  II  in  appropriate  field 

Level  III  in  appropriate  field  for 
CMO  Engineer  Lead 

Core 

DCMA  New/Advanced 

Engineering  Courses 

Commodity 

Appropriate  Licenses  or 
Certifications  for  Commodity 
(e.g.  Airframe  Powerplant  (A&P) 
License  for  Aero  Work) 

Specialty 

As  needed  (e.g.  EMI) 

DCMA 


Defense  Contract  Management  Agency 


Recommended  Training 
Path/Curriculum 


Training  Type 

Course(s) 

Developmental 

Leadership,  PBM,  Six  Sigma,  Predictive 
Analysis 

Professional  Certification 

Certified  by  Professional  Society  Aligned  with 
the  Individual’s  Career  Field  (as  desired) 

Additional  Recommended  Training 

Acquisition: 

DCMA  Integrated  Master  Schedule  Class 

DCMA  Systems  Engineering  Course 

BCF  102,203  (Earned  Value) 

LOG  101,204 

PQM  101,201 

TST102 

Enqineerinq:  e.q.,  TSNs,  NDT,  ANSI  Y-14.5M 
Geometric  Dimensioning  &  Tolerancing  (all 
as  needed) 

Product  Specific:  Determined  bv  DCMA 
Divisions  Based  on  Knowledge  Gap  Analysis 
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DCMA  Recommended  Path  Forward 


Defense  Contract  Management  Agency 


Establish  DCMA  HQ  Sys  Eng  Competency  Team 
•  Develop  Policy/Tools/Guidance 

*  Perform/Evaluate  Enterprise  Planning  to  include: 

•  Staffing/Organization 

•  Succession  Planning 

•  Appropriate  Skills  Matching 

•  Policy  and  Tools 

•  Training 

•  System  Engineering  Guide  Development 

•  Develop  Standard  Surveillance  Operating  Guide  (SSOG) 

•  Develop  Systems  Engineering  Influence  Guide 

•  Develop  Systems  Engineering  Evaluation  Guide  and 
associated  metrics 
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DCMA 


Defense  Contract  Management  Agency 


Standard  Systems  Operating  Guide 

(SSOG)  Outline 


Chapter  1 :  Concept  Development  Phase 

Perform  Program  Management  Oversight 

Perform  Engineering  Process  Reviews 

Evaluate  Engineering/Resource  Schedule  Estimates 

Evaluate  Program  Performance 

Perform  Engineering  Product  Examinations 


Chapter  2:  Technology  Development  Phase 

Perform  Program  Management  Oversight 

Perform  Engineering  Process  Reviews 

Evaluate  Engineering/Resource  Schedule  Estimates 

Evaluate  Program  Performance 

Perform  Engineering  Product  Examinations 
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DCMA 


Defense  Contract  Management  Agency 


Standard  Systems  Operating  Guide 

(SSOG)  Outline 


Chapter  3:  System  Development  and  Demonstration  Phase 
Perform  Program  Management  Oversight 
Perform  Engineering  Process  Reviews 
Evaluate  Engineering/Resource  Schedule  Estimates 
Evaluate  Program  Performance 
Perform  Engineering  Product  Examinations 


Chapter  4:  Production  and  Deployment  Phase 

Perform  Program  Management  Oversight 

Perform  Engineering  Process  Reviews 

Evaluate  Engineering/Resource  Schedule  Estimates 

Evaluate  Program  Performance 

Perform  Engineering  Product  Examinations 
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DCMA 


Defense  Contract  Management  Agency 


Standard  Systems  Operating  Guide 

(SSOG)  Outline 


Chapter  5:  Operations  and  Support  Phase 

Perform  Program  Management  Oversight 

Perform  Engineering  Process  Reviews 

Evaluate  Engineering/Resource  Schedule  Estimates 

Evaluate  Program  Performance 

Perform  Engineering  Product  Examinations 


Appendices:  DCMA  Systems  Engineering  Influence  Guide, 
Surveillance  Plan  Template,  Report  Results  and 
Recommendations,  Metrics,  and  Test  and  Evaluation 
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DCMA 


Defense  Contract  Management  Agency 


Engineering  Core  Process  IPT  - 

Membership 


Shaun  Lanham 
Larry  Cianciolo 
Thuyhong  Tran 
David  Kiewit,  P.E.,  JD 
Mike  Sheridan,  P.E. 
Bruce  Heim 
Terry  Taylor 
Capt  Mark  Fienhold 
Dr.  Ram  Sinha 


DCMAG-OCT 

DCMAM-OCT 

DCMAS-OCT 

DCMAM-OCT 

DCMAM-OCT 

DCMA  Boeing  LB 

DCMA  AIMO 

DCMA  SSO  Sunnyvale 

DCMA-HQ 
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NORTHROP  GRUMMAN 


Defining  the  future 


Abstract 
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Abstract 


NORTHROP  GRUMMAN 


It  is  a  common  practice  to  refer  to  applicable  documents  in  both  programmatic  and 
product-specification  documents  in  contracted  development.  The  practice  permits 
inclusion  of  a  vast  amount  of  lessons-learned  and  best  practices  can  be  referenced 
without  the  need  to  include  the  information  directly  in  the  document,  nor  to  maintain 
the  referenced  information.  Product  requirements  documents  often  specify  interfaces 
and  interoperability  characteristics  by  reference  to  interface  control  documents  included 
in  the  list  of  applicable  documents.  Benefits  accruing  to  the  product  from  the  use  of 
applicable  documents  are  reduced  overall  cost,  better  products  and  better 
interoperability.  Costs  accruing  to  the  product  development  effort  are  the  cost  of 
maintaining  visibility  on  changes  to  applicable  documents  outside  the  control  of  the 
Program  and  the  cost  of  verification  of  all  included  requirements. 

Experience  on  many  Programs  and  with  several  customers  has  shown  that  there  is  a 
wide  variation  in  the  manner  in  which  applicable  documents  are  incorporated  in  product 
specifications.  The  observed  differences  fall  into  several  broad  categories,  such  as:  the 
method  of  citation  of  applicable  documents;  the  difference  between  compliance  and 
reference  documents;  the  methods  of  referencing  the  documents  in  the  requirements 
statements;  and  the  approach  to  sub-tiering  of  the  applicable  documents. 

This  paper  will  discuss  the  different  approaches  to  utilizing  applicable  documents 
within  product  documents  and  the  issues  and  risks  that  arise,  illustrated  with  examples. 
Using  lessons  learned  across  the  program  and  customer  experience,  a  robust, 
standardized  approach  is  recommended  that  should  increase  the  benefit  of  using 
applicable  documents  while  reducing  the  cost. 
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Defining  the  future 


Context  and  Definitions 
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Context  and  Definitions 


NORTHROP  GRUMMAN 


•  Typical  specification  formats  utilized  in  US  Department  of  Defense 
contracting  provide  for  the  citation  of  applicable  documents. 

•  "Judicious  referencing  of  other  documents  in  specifications  is  a  valuable 
tool  that  eliminates  the  repetition  of  requirements  and  [eliminates  the 
repetition  ofl  tests  adequately  set  forth  elsewhere.  However, 
unnecessary  or  untailored  referencing  of  other  documents  can  lead  to 

increased  costs,  excessive  tiering,  ambiguities,  and  compliance  with 

unneeded  requirements."  (MIL-STD-961E,  4.19) 

•  Method  for  incorporating  lessons  learned  in  the  field 

•  Method  for  including  commercial  standards  and  practices 
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Context  and  Definitions  (Concluded) 


NORTHROP  GRUMMAN 


•  The  applicable  documents  are  of  two  types: 

-  Compliance  -  the  cited  document  contains  requirements  included  in  the 
citing  document  by  reference 

-  Reference  -  the  cited  document  provides  data  or  information  useful  in 
enhancing  the  understanding  of  the  citing  document 

•  Documents  can  be  referenced  in  product  specifications  and  in 
programmatic  documents  (e.g.,  Statement  of  Work).  This  presentation 
will  address  citation  in  product  specifications  only. 

•  Compliance  documents  can  be  found  in  functional,  performance, 
interface,  environmental  and  design  and  construction  requirements. 
Reference  documents  can  be  found  throughout  Sections  3,  4  and  5  and 
Appendices. 
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Applicable  Document  Utilization 


NORTHROP  GRUMMAN 


•  Various  standard  specification  formats  exist  (MIL-STD-490A,  MIL-STD- 
961E,  JSSG-2000A,  various  DIDs).  A  typical  format  is: 


1.0  Scope 

2.0  Applicable  Documents 
3.0  Requirements 

4.0  Verification  _ _ 

5.0  Packaging 
6.0  Notes 
lO.OAppendix 


Citations  of  all  documents 
cited  in  sections  3,  4  and 
5,  and  Appendices,  with 
full  attribution,  by  type 
and  source 

Document  citations  within 
specific  requirements 
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Areas  of  Confusion 


NORTHROP  GRUMMAN 


•  Citation  of  a  reference  document  as  a  compliance  document 

•  Citation  of  a  compliance  document  as  a  reference  document 

•  Unnecessary  citation  of  a  complete  document 

•  Incomplete  citations  in  Section  2 

•  Failure  to  state  precedence 

•  Failure  to  address  tiering 

•  Failure  to  flowdown  applicable  documents  to  subcontractors 
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•  The  author  has  found  no  discussions  of  the  use  of  applicable  documents 
in  the  literature. 

•  Some  examples  of  requirements  that  can  be  improved  are  given  in  this 
presentation.  There  is  no  intent  to  criticize  the  original  author,  as  there 
is  no  standard  way  to  handle  applicable  documents. 
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Defining  the  future 


Citation  in  Sections  3,  4  or  5 
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The  <System>  shall  limit  orbital  debris  generation  in  compliance  with 
NSS1740.14,  Guidelines  and  Assessment  Procedures  for  Limiting  Orbital  Debris. 


•  NSS1740.14  defines  parameters  for  assessing  a  design  to  determine  the 
minimization  of  orbital  debris  generation. 

•  This  could  be  a  programmatic  requirement  inadvertently  included  in  a 
product  specification. 

•  Alternatively,  one  can  derive  several  product  requirements  from  the 
Guidelines,  in  which  case  the  requirement  should  be  stated  as: 


The  <System>  shall  limit  orbital  debris  generation  using  NSS1740.14,  Guidelines 
and  Assessment  Procedures  for  Limiting  Orbital  Debris  for  guidance. 


•  NSS1740.14  should  then  be  listed  in  the  "Reference  Documents"  section. 
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The  <System>  shall  meet  MIL-STD-1472  for  all  1-g  human-machine  interfaces. 

The  <System>  shall  meet  NASA-STD-3000  for  all  micro-g  and  0-g  human- 
machine  interfaces. 


•  Both  requirements  are  valid  product  requirements. 

•  Both  MIL-STD-1472  and  NASA-STD-3000  contain  some  explicit 
requirements  ("shalls")  and  numerous  implicit  requirements  in  the  form 
of  design  guidance. 

•  It  has  been  estimated  that  each  of  the  documents  may  contain 
approximately  3000  requirements.  By  inclusion  of  the  entire  document, 
all  the  cited  requirements  will  have  to  be  verified. 

•  Should  only  cite  a  complete  document  if  the  intent  is  to  include  all  of  its 
requirements.  Otherwise,  cite  specific  sections. 
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The  <System>  shall  operate  after  temperature  and  humidity  diurnal  cycling 
During  transportation  and  storage  as  defined  in  MIL-STD-810F,  Method  507. 


•  MIL-STD-810F  is  not  a  product  requirements  document  -  it  addresses 
methods  for  environmental  testing. 

•  The  requirement  should  be  rewritten  as  a  product  requirement  for 
Section  3  and  a  verification  requirement  Section  4: 

The  <System>  shall  operate  after  temperature  and  humidity  diurnal  cycling 
During  transportation  and  storage  in  the  natural  environment  defined  in 
MIL-HDBK-310. 

Verification  of  the  <System’s>  operation  after  temperature  and  humidity  diurnal 
cycling  during  transportation  and  storage  using  Method  507  in  MIL-STD-810F. 


•  Both  MIL-STD-810F  and  MIL-HDBK-310  should  then  be  referenced  in  the 
"Reference  Documents"  section. 
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The  <System>  shall  operate  during  and  after  exposure  to  rain  at  a  rate  of  100 
millimeters  (mm)  per  hour  for  a  1  hour  duration  at  +24  degrees  Celsius  (C)  and 
with  a  64-knot  wind  as  defined  in  MIL-HDBK-310. 

The  <System>  shall  be  sealed  to  prevent  water  incursion  as  defined  in 
MIL-STD-810F,  Method  506,  Section  2.2.2b  (Water  Tightness). 


•  The  first  requirement  is  a  good  example. 

•  The  second  requirement  should  be  rewritten  as  a  product  and  a 
verification  requirement: 


The  <System>  shall  be  sealed  to  prevent  water  incursion  (Water  Tightness). 

Verification  that  the  <System>  is  sealed  to  prevent  water  incursion  shall  be 
conducted  using  Method  506,  Section  2.2.2b  of  MIL-STD-810F. 
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2.0  APPLICABLE  DOCUMENTS 

2.1  APPLICABLE  DOCUMENTS 

The  following  documents  are  applicable  to  the  <ltem>  requirements: 

2.2  REFERENCE  DOCUMENTS 

The  following  are  reference  documents: 


•  The  first  problem  is  the  use  of  the  same  title  for  sections  2.0  and  2.1. 
Section  2.1  should  be  called  "COMPLIANCE  DOCUMENTS". 

•  The  introductory  text  in  Section  2.1  places  no  restrictions  on  the  cited 
documents  -  if  a  document  is  updated  at  some  point  during  the  product 
life  cycle,  then  the  product  must  be  updated  to  agree  with  the 
compliance  documents. 
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2.0  APPLICABLE  DOCUMENTS 

The  documents  listed  in  this  section  are  specified  in  sections  3,  4,  or  5  of  this 
document.  This  section  does  not  include  documents  cited  in  other  sections  of 
this  document  or  recommended  for  additional  information  or  as  examples. 
While  every  effort  has  been  made  to  ensure  the  completeness  of  this  list, 
Document  users  are  cautioned  that  they  must  meet  all  specified  requirements 
of  Documents  cited  in  sections  3,  4,  or  5  of  this  document,  whether  or  not  they 
are  listed  here.  Failure  to  include  a  cited  document  in  this  section  does  not 
mean  that  it  is  not  included  in  this  Document.  Inclusion  of  a  document  in  this 
section  without  a  citation  in  the  text  does  not  include  that  document  in  this 
Document. 

2.1  APPLICABLE  DOCUMENTS 

The  following  documents  of  the  exact  revision  and  date  listed  below  form  a  part 
of  this  specification  to  the  extent  specified  herein. 

2.2  REFERENCE  DOCUMENTS 

The  following  documents  of  the  exact  revision  and  date  listed  below  are 
referenced  herein. 


•  The  statements  force  exact  attribution  of  the  applicable  documents 
and  ensure  that  any  update  to  an  applicable  document  will  force  a 
formal  explicit  review  and  possible  change  to  the  specification. 
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•  The  format  for  Sections  2.0,  2.1  and  2.2  shown  on  the  previous 
chart  are  the  "short"  form. 

•  MIL-STD-490  and  MIL-STD-961  recommend  listing  the  documents 
within  Sections  2.1  and  2.2  by  source. 

•  See  Section  5.7.2  -  5.7.3  of  MIL-STD-961E  for  an  example  of  the 
"long"  form.  Note  that  it  makes  no  distinction  between  compliance 
and  reference  documents. 
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•  A  standard  format  is  to  reference  the  documents  (compliance  and 
reference)  in  a  tabular  format,  as  follows: 


MIL-DTL-15090D  Detail  Specification,  Enamel,  Equipment,  Light  Gray,  (Navy  Formula  No.  Ill),  Department  of 

6  November  1996  Defense 


MIL-STD-1399-300A  Military  Standard,  Interface  Standard  for  Shipboard  Systems,  Section  300A,  Electric  Power, 

Notice  1  Alternating  Current,  Department  of  Defense 

11  March  1992 


MIL-STD-1472F  Department  of  Defense  Design  Criteria  Standard,  Human  Engineering,  Department  of  Defense 

Notice  1 

5  December,  2003 


•  Decide  if  the  citation  in  the  requirement  statement  should  have  the 
full  attribution,  or  just  the  base  number  (i.e.,  MIL-STD-130M  or  MIL- 
STD-130).  The  full  attribution  must  be  provided  in  Section  2. 
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Defining  the  future 


Precedence 
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•  There  can  be  conflicts  between  the  cited  documents  and  requirements  in 
the  citing  specification. 

•  Add  a  subsection  to  Section  2  with  the  following  text: 

2.X  Order  of  precedence 

In  the  event  of  a  conflict  between  the  text  of  this 
specification  and  the  references  cited  herein,  the  text 
of  this  specification  takes  precedence.  Nothing  in  this 
specification,  however,  supersedes  applicable  laws  and 
regulations  unless  a  specific  exemption  has  been 
obtained. 

Quoted  from  JSSG-2000A,  paragraph  2.5. 


Cleared  for  Public  Release,  Control  No.  08-103,  dtd.  9/30/08 


23 


NORTHROP  GRUMMAN 


Tiering 
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•  Control  of  document  tiering  has  become  a  primary  way  of  controlling 
contractual  applicability  of  cited  documents.  Care  must  be  taken  to 
ensure  that  each  cited  document  is  appropriate  to  the  first-tier 
references  or  compliance  documents  (including  those  references  or 
compliance  documents  cited  in  the  contract,  which  themselves  would 
become  first-tier  references  or  compliance  documents  and,  thus,  their 
second  tier  would  become  contractually  applicable  as  well). 

•  Exceptions  to  tiering  applicability  are  generally  defined  by  DoD  policy. 
For  example,  in  the  Perry  memo  previously  cited,  the  direction  on 
tiering  of  specifications  and  standards  includes,  "Approval  of  exceptions 
may  only  be  made  by  the  Head  of  the  Departmental  or  Agency 
Standards  Improvement  Office  and  the  Director,  Naval  Nuclear 
Propulsion  for  specifications  and  drawings  used  in  nuclear  propulsion 
plants  in  accordance  with  Pub.  L.  98-525  (42  U.S.C.  fl7158  Note)." 

Based  on  JSSG-2000A,  paragraph  2.4. 
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•  Cited  applicable  documents  can,  themselves,  cite  additional  applicable 
documents  which  can,  also,  cite  applicable  documents: 
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•  Need  to  include  a  tiering  statement  such  as: 

When  the  <item>  specification  is  directly  referenced  in  the 
contract,  it  is  a  first-tier  specification  and  is  applicable. 
Documents  referenced  in  the  first-tier  specification  are 
applicable  as  follows: 

a.  Second  Tier  -  All  documents  directly  referenced  in  the 
first-tier  specification  are  only  applicable  to  the  extent 
specified. 

b.  Lower  Tier  -  All  documents  directly  referenced  in  second- 
or  lower-tier  documents  are  for  guidance  only  unless 
otherwise  directed  by  the  contract. 

Based  on  JSSG-2000A,  paragraph  2.4. 
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Defining  the  future 


Flowdown  to  Subcontractors 
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•  The  specifications  provided  to  the  subcontractors  for  the  items  that  they 
are  to  provide  should  also  contain  a  Section  2,  and  the  flowed-down 
requirements  should  cite  the  portion  of  the  applicable  document  in  the 
parent  document  that  corresponds  to  the  flowed-down  requirement 
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A  recognized  systems  engineering  best  practice  is  early  development  of 
operational  concepts  during  system  development  and  documentation  of  those 
operational  concepts  in  one  or  more  Operational  Concept  Documents.  Recognizing 
this  best  practice.  United  States  Department  of  Defense  and  NASA  standard 
procedures  require  that  information  relating  to  system  operational  concepts  be 
prepared  in  support  of  the  specification  and  development  of  systems.  I  n  the  past, 
the  DoD  has  published  Data  Item  Descriptions  (Dl  Ds),  and  NASA  has  published 
Data  Requirements  Documents  (DRDs),  which  describe  the  format  and  content  of 
the  information  to  be  provided. 

The  abstract  model  of  the  concept  of  operations  is  focused  on  operations,  with 
reference  to  specific  systems  only  as  needed.  A  definition  of  “operational  concept" 
has  not  been  found.  For  the  purposes  of  this  presentation,  it  will  be  taken  to  mean 
an  abstract  model  of  the  operations  of  a  specific  system  or  group  of  systems, 
usually  developed  as  part  of  the  acquisition  process  and  used  throughout  the 
design,  development,  test  and  evaluation  (DDT&E)  phases  of  the  system  life  cycle. 

While  various  Government  standards  require  the  generation  of  operations 
concept  information  and  Data  Item  Descriptions  (Dl  Ds)  are  available,  little 
information  is  typically  provided  which  clearly  describes  the  manner  in  which  an 
OCD  should  be  used  in  support  of  a  system  development.  Few  guidelines  exist 
regarding  which  information  is  most  useful,  how  to  develop  that  information, 
which  developer  and  customer  personnel  should  participate,  or  how  to  document  it 
have  been  provided. 

This  presentation  will  address  the  nature  of  the  operations  concept,  how  it  is 
developed  and  by  whom,  and  how  it  is  used  in  the  development,  deployment, 
operations  and  support  of  a  new  or  upgraded  system. 
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•  What  is  an  OpsCon 

•  Relationships  with  Use  Cases  and  DoDAF 

•  OpsCon  Preparation 

•  Scenarios 

•  Requirements  Definition  from  the  Operational  Concept 

•  Use  of  the  Operational  Concept  for  Validation 

•  References 

•  Author  Biography 
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•  Concept  of  Operations:  A  verbal  and  graphic  statement,  in  broad  outline,  of  an 
enterprise's  assumptions  or  intent  in  regard  to  an  operation  or  series  of 
operations.  The  concept  of  operations  frequently  is  embodied  in  long-range 
strategic  plans  and  annual  operational  plans.  I  n  the  latter  case,  the  concept  of 
operations  in  the  plan  covers  a  series  of  connected  operations  to  be  carried  out 
simultaneously  or  in  succession.  The  concept  is  designed  to  give  an  overall 
picture  of  the  enterprise  operations.  It  is  afso  called  the  CONOPS. 

-  (Based  on  Department  of  Defense  Dictionary  of  Military  and  Associated  Terms,  J  P  1-02, 
12  April  2001  (as  amended  through  23  March  2004) 

•  Operational  Concept:  A  verbal  and  graphic  statement  of  an  enterprise's 
assumptions  or  intent  in  regard  to  an  operation  or  series  of  operations  of  a 
system  or  a  related  set  of  systems.  The  operational  concept  is  frequently 
developed  as  part  of  a  system  development  or  acquisition  program.  The 
operational  concept  is  designed  to  give  an  overall  picture  of  the  operations  using 
one  or  more  specific  systems,  or  set  of  related  systems,  in  the  enterprise's 
operational  environment  from  the  users'  and  operators'  perspective.  1 1  is  also 
called  the  OpsCon.  1 1  is  defined  in  an  Operational  Concept  Document. 

•  These  definitions  will  be  used  in  this  presentation.  They  are  arbitrary  -  define 
the  terms  for  your  Program  at  the  beginning  of  the  Program  and  use  them 
consistently. 
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Enterprise  Level 


System  Modification 
Or  Upgrade 


System  Level 
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Problem  description 


Problem  description 


Solution  description 


Increasing 

detail 

v 


Pre-Proposal 
Operational 
Concept 


•Prepared  by  customer  /  users  /  operators 

•User’s  /  Operator’s  viewpoint 

•Basis  for  the  selection  of  MoEs 

•Basis  for  selection  of  standards  of  acceptance 


Development 
Operational 
Concept 

Production 
Concept 

Deployment 
Concept 

Support 
Concept 


Disposal 

Concept 


•Prepared  by  customer  /  users  / 
operators  and  developer 
•User’s  /  Operator’s  viewpoint 
•Describes  intended  behaviors 
•Basis  for  verification  and 
validation  planning 
•Basis  for  design  of  End  Products 
and  Enabling  Products 
•Basis  for  number  of  units, 
availability,  deployment  location 
decisions 

•Basis  for  evaluation  for  future 
change  requests 
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•  The  Operational  Concept  provides  contextual  information  for  the  development 
of  the  requirements  and  tne  system  -  details  of  the  intended  use  and  benefits  of 
the  system. 

•  The  Operational  Concept  provides  the  basis  for  the  system  validation. 


System  Concept 
Development  - 
Operational  Concept 
Document 


Requirements 

Development 


Production, 
Deployment, 
Operation,  Support, 
Disposal 
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•  The  Use  Case,  part  of  the  Unified  Modeling  Language  (UML),  and  now 
part  of  the  System  Modeling  Language  (SysML)  as  well,  is  limited  in 
scope  relative  to  an  Operational  Concept  Document  (OCD). 

•  Use  Case  analysis  is  extremely  useful  in  elicitation  of  operator, 
maintainer  and  user  requirements. 

•  The  Use  Case  does  not  discuss  the  operational  context  or  environment, 
for  example. 

•  Another  valuable  feature  of  Use  Cases,  which  can  be  included  in  the 
OCD,  is  the  Use  Case  scenario  pre-  and  post-conditions. 

•  The  set  of  Use  Case  descriptions  and  the  scenarios  can  be  included  in 
the  OCD,  but  they  do  not  form  a  complete  OpsCon 

•  The  use  of  Use  Cases  in  the  OpsCon  should  be  carefully  considered  and 
balanced  with  the  use  of  narrative  and  graphics  for  the  remainder  of 
the  OCD.  A  collection  of  Use  Cases  and  Scenarios  can  not,  in  itself, 
constitute  an  OCD. 
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OpsCon  and  DoDAF  OV-1 
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•  OV-1,  High-Level  Operational  Concept  Graphic 

•  The  High-  Level  Operational  Concept  Graphic  describes  a  mission  and 
highlights  main  operational  nodes  (see  OV-2  definition)  and  interesting  or 
unique  aspects  of  operations.  It  provides  a  description  of  the  interactions 
between  the  subject  architecture  and  its  environment,  and  between  the 
architecture  and  external  systems.  A  textual  description 
accompanying  the  graphic  is  crucial.  Graphics  alone  are  not 
sufficient  for  capturing  the  necessary  architecture  data. 

•  The  Operational  Concept  Document,  which  documents  the  Operational 
Concept,  is  ideal  to  provide  the  textual  description  for  the  OV-1  Graphic. 
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•  There  are  two  documents  that  provide  guidance  in  the  development  of 
Operational  Concepts  and  their  documentation: 

-  ANSI  /  Al  AA  G-043- 1992,  Guide  for  the  Preparation  of  Operational  Concept 
Documents,  American  I  nstitute  of  Aeronautics  and  Astronautics,  Washington,  D.C., 
J  anuary  22, 1993  (being  updated) 

-  FHWA- HOP-07-001,  Developing  and  Using  a  Concept  of  Operations  in 
Transportation  Management  Systems,  Federal  Highway  Administration,  August 
2005 

•  Note  that  the  report  is  actually  addressing  the  Operational  Concept  as  defined 
on  Chart  4 

•  There  are  two  additional  documents  that  provide  outlines  and  a 
discussion  of  the  contents  of  an  OpsCon 

-  I EEE  Standard  1362, 1 EEE  Guide  for  I  nformation  Technology  -  System  Definition  - 
Concept  of  Operations  Document,  19  March  1998 

-  I  SO  14711:2002(E),  Space  systems  -  Unmanned  mission  operations  concepts  - 
Guidelines,  2002 

•  US  Government  guidance  documents  are  listed  in  the  References. 
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Stakeholders 


NORTHROR  GRUMMAN 


Operators,  Maintainers  and  Users  should  be  the  authors  of  the 
OpsCon.  In  the  event  that  they  have  not  developed  an  OpsCon, 
then  the  developer  community  should  develop  it  in  cooperation 
with  the  operators,  maintainers  and  users.  It  is  possible  that 
some  stakeholders  are  not  available  (e.g.,  regulators),  in  which 
case  the  authors  must  work  with  stakeholder  surrogates. 
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When  to  Write  the  OpsCon 
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•  The  OpsCon  should  be  written  before  or  during  the  concept  development 
phase. 

•  The  OpsCon  should  be  updated  throughout  the  product  lifecycle,  at  major 
milestones,  to  support  the  Program  phase: 

-  System  Development 

-  Production 

-  Deployment 

-  Operations  and  Support 

-  Diqnnul 
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Users  of  the  OpsCon 
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•  Operators  /  Maintainers  /  Users 

•  Systems  Engineers  and  Architects 

•  System  I  mplementers 

•  Acquirers 

•  Testers 

•  Regulators 

•  I  n  short,  the  stakeholders 
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Types  of  OpsCon 
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•  Operations  Concept  -  describes  the  way  the  system  works  from  the 
operator's  perspective. 

•  Production  Concept  -  describes  the  way  the  system  will  be  manufactured. 

•  Deployment  Concept  -  describes  the  way  the  system  will  be  delivered  and 
installed. 

•  Support  Concept  -  describes  the  desired  support  infrastructure  and 
manpower  considerations  for  maintaining  the  system  after  it  is  deployed. 
This  includes  specifying  equipment,  procedures,  facilities  and  operator 
training  requirements. 

•  Disposal  Concept  -  describes  the  way  the  system  will  be  removed  from 
operation  and  retired. 

-  Based  on  the  I NCOSE  Systems  Engineering  Handbook 
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Environments 
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•  The  OpsCon  describes  the  various  environments  in  which  the  system  will 
be  deployed,  operate  and  be  maintained: 

-  Physical 

•  Natural 

•  Induced 

•  Self- induced 

•  Threat 

•  Cooperative 

-  Political 

-  Social 

-  Economic 

-  etc 
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•  1.  Scope 

•  2.  Referenced  Documents 

•  3.  Background  I  nformation 

•  4.  Existing  Systems  and  Operations 

•  5.  Operational  Overview 
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•  Appendix  A:  Acronyms,  Abbreviations  and  Glossary 

•  Appendix  B:  System  Operational  Scenarios 
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Scenarios 
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•  Scenarios  consist  of  both  a  textual  and  graphical  description  of  a  single 
end-to-end  thread  of  operation  of  the  system 

•  Need  to  have  the  major,  critical,  normal  operational  scenarios  (“happy 
day")  described 

•  Need  to  have  the  important  off-design  or  degraded- mode  operational 
scenarios  described  as  well  (“rainy  day“) 

•  Some  examples  of  the  graphical  representation  follow 

-  Typically,  would  use  Functional  Flow  Block  Diagrams,  Enhanced 
Functional  Flow  Block  Diagrams,  Activity  Diagrams  or  Sequence 
Diagrams 

-  Have  used  a  non-standard  notation  for  simplicity 
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Example  Scenario  -  "Happy  Day" 
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Example  Scenario  -  "Rainy  Day  1" 


NORTHROR  GRUMMAN 


Service 

Aircraft 


21 


Cleared  for  Public  Release,  Control  No.  08-104,  dtd.  9/  30/  08 


Example  Scenario  -  "Rainy  Day  2" 


NORTHROP  GRUMMAN 


22 


Example  Scenario  -  "Rainy  Day  3" 
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Developing  Requirements  from  the  Operational 
Concept 
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■  The  Operational  Concept  defines: 

•  the  context  of  the  product,  leading 
to  a  definition  of  its  boundaries  and 
the  external  interfacing  and  inter¬ 
operating  systems,  leading  to 
identification  of  interface 
requirements; 

•  the  normal  and  other  operational 
environments,  leading  to  definition 
of  the  environmental  requirements; 

•  scenarios  of  normal  and  degraded 
operations,  from  which  Product 
functionality  can  be  derived; 

•  and  scenarios  showing  a  "day  in  the 
life"  of  the  product  which  help  to 
develop  the  logistics,  maintenance 
and  support  requirements,  and 
identify  the  personnel  requirements 
for  the  product  operator. 

Cleared  for  Public  Release,  Control  No.  08-104,  dtd.  9/  30/  08 


24 


Use  of  the  Operational  Concept  for  Validation 


NORTHROR  GRUMMAN 


■  Requirements  Validation  is  the 
task  of  showing  that  the  Product 
as  developed  satisfies  the 
customer  needs  in  its  intended 
operational  environment. 


•  The  Operational  Concept  provides: 

-  A  summary  of  the  customer  needs 

-  A  description  of  the  normal  and  other  operational  environments 

-  Various  operational  scenarios  that  can  be  used  to  define  validation 
test  procedures 

-  Scenarios  of  degraded  operations 
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Using  Performance-Based  Earned  Value®  for 
Measuring  Systems  Engineering  Effectiveness 
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Bojan  Zlicaric 

The  Boeing  Company 


NDIA  SE  Conference  -  20  October  2008 


Outline 

■  Performance-Based  Earned  Value® 

■  SE  Effectiveness 

■  SE  Metrics  Architecture 

■  Example  Metrics  for  Requirements 


The  Scope  of  Earned  Value  is  Limited 


■  *ANSI/EIA-748B,  3.8 

■  “Earned  value  is  a  direct  measurement  of  the 
quantity  of  work  accomplished.  The  quality,  and 
technical  content  of  work  performed  is 
controlled  by  other  processes.”  [emphasis 
added] 

■  Need  another  method  to  assess  quality  of 
work  accomplished 


*  “Standard  for  Earned  Value  Management  Systems” 


Easy  PBEVSM  Example 


■  Task:  wash  windows 

■  Desired  outcome:  clean  windows 

■  Quality  measure:  cannot  see  anything  on  window 
surface  (no  distortion  or  obscuration  of 
reflections) 

■  Earned  Value:  Window  was  washed 

■  “I  washed  the  window” 

■  PBEVSM:  Window  is  clean 

■  “But  it’s  not  clean”  -  PBEVSM  less  than  EV 

■  Difference  (PBEVSM  -  EV)  =  “Unearned  value”  = 
Quality  criteria  for  the  product  delivered  by  the 
activity,  or  the  cost  of  rework 


What  is  Quality? 


■  “Quality  is  conformance  to  requirements” 
(Crosby,  “Quality  is  Free”,  1979) 

■  Therefore,  “quality”  of  work  accomplished  is 
composed  of 

■  Inherent  quality  of  work  product  (conformance  to  work 
product  standards,  e.g.,  specs,  drawings,  plans,  reports) 

■  Conformance  of  work  product  to  technical  requirements 

associated  with  the  system  (e.g.,  design  satisfies 
requirements) 


SE  Quality  Example  -  Specifications 


■  A  major  SE  work  product  is  a  specification 
containing  all  requirements  for  a  system 

■  Requirements  Specification  Quality  -  2  parts 

■  Specification  structure  and  syntax 

"  Conforms  to  template  standards  (quality  of  specification) 

■  Completeness,  outline,  format 

■  Requirements  are  well-stated  (quality  of  requirements) 

■  Clarity,  verifiability,  etc. 

■  Specification  content 

■  System  described  satisfies  user  needs  and/or  contract 
requirements,  e.g.,  weight,  speed,  availability,  etc. 


SE  Effectiveness 


"  “Effectiveness”  is  an  ability  to  produce  the  needed  result 
using  the  committed  resources 

■  Resource  commitments  based  on  planning 

■  EV  measures  execution  vs.  plan 

■  Resource  utilization:  money,  people,  facilities,  time 

■  What  are  the  “needed  results”  or  products  of  SE? 

■  Specific  SE  work  products 

■  Program  outcomes 

■  Cost  -  Budgeted  cost 

■  Schedule  -  Committed  schedule 

■  Technical  Performance  -  Systems 
satisfying  requirements  and  needs 

>  Leads  to  PBEVSM* 


*PBEV  and  Performance-Base  Earned  Value  are  registered  trademarks  of  Paul  Solomon 


SE  Effectiveness  Decomposition 


■  Define  contributors  to  SE  Effectiveness 
"  Leads  to  SE  Metrics  Architecture 


Three  contributing  streams 
■  Product  Quality  -  Satisfying  needs  and  requirements 
"  Cost  and 


Schedule 


Collectively  measured  by  Earned  Value 


Planning  (basis  for  product  definition  and  EV) 


■  Essential  elements 

■  Work  product  quality  and  completeness  -  fitness  for  use  by 
downstream  “customer” 

■  Timeliness  -  available  when  needed 

■  Defined  by  coordinated  schedule;  measured  by  EV 


0-' 


Using  PBEVSM  for  SE  Effectiveness 

rt  - 


■  Work  definition  -  IMP/IMS 

■  Define  work  products  for  every  scheduled  activity  (evidence  of 
completion) 

•  Plans,  requirements,  design,  interfaces,  verification 

■  Define  objective  quality  standards  for  work  products 

■  Define  technical  content  requirements  for  work  products 

■  Progress  assessment 

Ik  -  ■  Value  is  earned  (EV)  based  on 

■  Satisfying  work  product  quality  standard 

■  Satisfying  technical  requirements  associated  with  work  product 

1  ■  Technical  maturity  per  plan  -  %  of  planned  TPM  achieved 
(Solomon) 

■  “Unearned  value”  is  cost  of  rework:  the  work  not-yet- 
accomplished 


Decomposition  of  Compliance  of  Design  with  Requirements 


■  Measure  Quality  and  Completeness  of 

■  Design  Analysis  and  Verification  (Compliance) 


Technical  Compliance  Metric 


At  each  major  review,  assess  %  requirements  for  which  design  is 
compliant,  with  associated  risk  level  of  non-compliance* 


Requirements  Compliance  Status 
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Requirements  Quality  Assessments  (RQA) 

■ 


Assess  quality  of  requirements  vs.  objective 
quality  standard 
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■  EV  alone  is  inadequate  to  assess  technical 
progress 

■  Program  goals  include  satisfying  cost, 
schedule,  technical  requirements 

■  PBEVsm  offers  a  method  to  integrate  these 


■  Architecture  of  SE  measures  enables 
decomposition  and  allocation  of  PBEVSM 
contributors  to  measurements  of  common  SE 
work  products 
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Advanced  Systems  &  Supportability  Engineering  Technology  and  Tools 


□  Challenge:  Incorporate  Operational  Realism 
Early  in  Life  Cycle  -  Today! 

■  What  products  &  when  in  life  cycle 

■  Two  current  projects  that  address  challenge 

□  Double  Helix  Methodology  involving 
CONOPS/Technology  Trade-offs 

□  The  Methodology  Captures  Key  Acquisition 
Information 

■  Inputs  for  TES,  TDS,  and  TDS 

■  Use  Case  involvement 

■  Incorporating  operations  in  Test  Architecture 

□  Lessons  Learned  at  ASSETT 

□  Summary  and  Conclusions 

□  Q&A 


22  October  2008 


Slide  2 


SSETT  Challenge:  Operational  Realism  Early  in  Cycle 


Advanced  Systems  &  Supportability  Engineering  Technology  and  Tools 


□  Challenge  -  NDIA  SE  Division  DT&E  Report  [April  2008]: 

■  Finding:  Operational  realism  is  often  not  included  or 
detailed  in  the  earliest  phases  of  acquisition,  such  as  during 
generation  of  the  CONOPS,  ICD,  TDS,  and  TES 

■  Recommendations: 

□  Operational  realism  must  be  given  due  diligence  during  the 
generation  of  the  CONOPS,  then  flowed  into  the  ICD,  TDS,  and 
TES 

□  CONOPS  should  have  iterative  updates  beginning  when 
technology  constraints  are  identified... 

□  These  updates  need  to  flow  into  the  ICD,  TDS,  and  TES  or 
their  respective  follow-on  documentation. 

□  In  the  following  charts,  the  approach  being  done 
by  a  small  business,  ASSETT,  is  shown  to  be 
accomplishing  the  recommendations. 
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Operational  Realism  and  CONOPS 


Advanced  Systems  &  Supportability  Engineering  Technology  and  Tools 


□  Operational  Realism  -  The  tasks  and  activities,  operational 
elements,  and  information  exchanges  required  to  conduct 
operations 

■  DODAF  Operational  View  in  a  System  Architecture 

□  Includes  high  level  operational  concepts  [e.g.  CONOPS], 

□  Operational  activities  sequence  and  tinning  descriptions 

□  Activity  and  Logical  data  models 

■  Trade-offs  between  operations  and  technologies 

□  CONOPS  -  A  Concept  of  Operations  is  defined  as  a  description 
of  how  a  set  of  capabilities  may  be  employed  to  achieve 
mission  objectives  or  a  particular  end  state  for  a  specific 
scenario 

■  A  CONOPS  for  critical  mission  segments  should  be  in  place 
for  all  mission  scenarios 

■  Currently,  a  CONOPS  is  not  updated  for  a  platform  even 
though  a  technology  improvement  is  installed  or  a  new 
capability  made  available. ..CONOPS  should  change 
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Operational  Realism  Products  Evolve  Early  in 
Acquisition  and  SE  Process 
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TRAD 


UDP 


Conceptual 

System  Design 

Specification 

System  Baseline  I  Component/  Application  Design  Baseline 

SRRBasel'ne  PDR  CDR 


Acceptance 

Development  jest 

Test  Baseline  |  Production  Baseline 
TRR  PRR 


Inception 


Transition 

Maintenance 

PDR  Package 


CDR  Package 
i  User  Needs  &  Technology  Opportunities  ! 


HW  &  SW  Production  Test 


Concept 

Technology 

System  Development  &  Demonstration 

Production  & 

Refinement 

Development 

Deployment 

ACQ 

FWK 


SE 


DE 


•  ICD  -  Initial  Capabilities  Document 

•  TDS  -  Technology  Development  Strategy 

•  TES  -  Test  &  Evaluation  Strategy  [for  TEMP] 


System  Requirements 
Specification  ("A") 

System  Use  Cases 

System  Architecture 


Data  Architecture 


Test  Strategies 
High  Level  Design 
Test  Architecture 
Subsystem 
Requirements 
Specification  ("B") 


Software 

Architecture 


V 


System  Iteration  Independent  Test  Plans, 
Procedures,  Test  Data 

System  Requirements  Traceability  Verification 
Matrix  (SRVM) 


Developed  and  Unit  Tested  Software 

Developed  and  Unit  Tested  Hardware 
Hardware  and  Software  Unit  Test  Plans 


Test  Results 
Test  Reports 


'  Computer  SW 
End  Items 


SW  Delivery 
Acceptance 


V 
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Two  Current  ASSETT  Projects  Address 
Operational  Realism 


□SBIR  N05-149  Combat  System  of  the  Future 
Non-Traditional  View  of  the  Submarine 
□Provide  the  Basis  for  Ship  Design 

■  HSI  Impacts  Manning  Reduction  that  Drives  Stores, 
Accommodations,  and  Supplies 

■  Maximum  Use  of  Technology  that  Drives  Power,  Cooling, 
Volume,  and  Footprint  Requirements 

□identify  Changes  in  CONOPS  and  Training 

■  Allow  CONOPS  to  Change  as  a  Function  of  Technology 

■  Develop  Confidence  in  New  Analysis  Tools  and  Automation 

□ONR  Capable  Manpower  Initiative 

■  BAA  007-013  -  Improved  Manning  and  Optimized 
Personnel  (IMOP) 

□Top-Down  Approach  to  Estimating  the  Manning  Requirements 
for  a  Platform 

□Searches  for  an  Optimum  Manning  Solution  Among  Number 
of  Operators,  System  Resource  Requirements  and  Mission 
Tasking 


System  Resources 


22  October  2008 


Slide  6 


Our  CSoF  Methodology  Incorporates 
Operational  Concepts  and  Technology 
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u  commander  s  £ 
Cognitive  w 
Requirements 


Double  Helix  Approach 
Leveraged  from  the  DARPA 
Command  Post  of  the 
Future  [CPoF]) 


•  Technology  (Helix  1)  and  CONOPS  (Helix  2  -  Operator's  View)  Evolve  and  Over 
the  Life  Cycle  of  the  System  (Concept  Through  Disposal) 

■  Understanding  the  Operator's  View  and  What  Is  Needed  for  Effective  Decision 
Making  Is  Necessary  In  Order  to  Apply  New  Technologies  Effectively 

■  Conversely,  the  Operator  Needs  to  Be  Made  Aware  of  New  Technologies  and  How 
They  May  Impact  His  Decision  Making 

■  The  Blue  Vertical  Bars  Represent  Points  in  Time  When  an  Exercise  Is  Run  to 
Determine  if  Changes  in  Technology  or  CONOPS  Would  Enhance  the  Operator's 
Ability  to  Make  Accurate  Decisions.  Typically  this  Exercise  Is  Via  the  Web  or  Video 
Conferencing 

■  The  Orange  Vertical  Bars  Represent  Point  in  Time  When  Actual  Experiments  Are  Run 
to  Analyze  the  Benefits  of  New  Technology  or  Changes  in  CONOPS. 
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Mission  Driven  Design 


Advanced  Systems  &  Supportability  Engineering  Technology  and  Tools 


□  Top  Down  Approach  that  Focuses  On  Ensuring  Mission  Success 

Develop  Mission  Scenarios 

Determine  Alternatives  for  Presenting  Information  to  Operator  (s) 

■  The  ASSETT  Team  Executes  Web  Exercises,  Interviews,  &  Team  Experiments 

□  SMEs  Identify  the  Decision  Points  and  the  Information  Required 

■  System  Engineers  Determine  Technologies  and  Capabilities  Necessary  to 
Provide  the  Required  Information 
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Operations  &  Technology  Analyses  Driven  by 
Missions  to  Optimize  Manning  -  Data  Gathering 
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•  Personnel/  Roles  of 
Current  Submarines  _ 
(CSoF) 

•  An  initial  CONOPS 
identifying  Candidate 
Operators  to  Eliminate 
or  Redefine 

•  Manning  Goal  of 
current  crew  size 

•  Zero-based  double 
helix  manning 
analysis  (CSoF) 


Tasks  are  defined  in 
the  IMOP  Manning 
Model  with  Trade-offs 
between  Operators  & 
Resources 


(System)  Resources 


Initial 

Technology 

Candidates 


PHOTONICS 
RADAR 


An  Initial  Basis  for  a  Resource  Hierarchy 


OBU  LWR  /  COMMS 


NON-TACTICAL 
TOTAL  DATA  PR0C  ESSI NG 

EXTERIOR  SH|p 
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Building  with  the  Combat  System  of 
the  Future  (CSoF)  Process 


MiSSiOIISI  (Representative 
mission  phases  from  the  CSoF 
Operational  Scenarios:  Port  Egress, 
Submerged  Transit,  and  Intelligence 
Surveillance  and  Reconnaissance 
(ISR)) 


Proceeding  Ahead: 

Refocus  Data  Gathering 
to  narrow  search  and 
expand  data  attributes  for 
modeling 


Objective:  Formulate  data  to 
define  operational  decisions 
&  capture  technology  for 
TDS  &  ICD  inputs 


Operational  View 
[DODAF] 

Identifies  What  Needs  to  Be 
Done  and  Who  Does  It 


Operational 

Elements 


WDX  Information  Flow 

SME  Interviews 
Step  1 

Decision  Information 
Required/Task  Timeline 

CONOPS  (Re-thought  NWP  for  every 
activity  for  each  mission) 

Decision  Hierarchies,  Timelines 


pV,#  e* 


Resources 
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Our  Process  Would  Drive  Operational 
Needs  Into  the  TES,  ICD,  and  TDS 


Operational  Scenarios, 
Tasks,  &  Decision 
Information 


Operator  HSI  Design 


(Page  1  of  2) 


Test  and  Evaluation  Strategy  (TES)  Objectives 

(Defense  Acquisition  Guidebook  -  Section  9.3.1) 

■  Verify  SE  Process 

■  Event  driven  T&E  Strategy 

□  Assess  technical  progress  against  critical  technical 
parameters  (CTP) 

□  Determine  readiness  of  operational  testing  -  an  OT&E 
entrance  criteria 

□  Assess  command,  control,  communications,  and  ISR  to 
ensure  interoperability  will  represent  stressed  OT&E 
scenarios 

■  How  stress  system  to  at  least  the  limits  of  the 

Operational  Mode  Summary/Mission  Profile 


Technology  Development  Strategy  (TDS)  Objectives 

(Defense  Acquisition  Guidebook  -  Section  2.2.1) 


■  Focus  is  on  the  Technology  Development  Phase  activities  (TDP) 

■  Strategy  to  manage  R&D 

■  Description  of  1st  technology  demonstration 

■  Test  Plan: 

□  How  1st  technology  spiral  demo  will  be  evaluated 

□  Focus  on  evaluation  of  technologies  being  matured  during  TDP  and  signals  end 
of  Milestone  A 
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Our  Process  Would  Drive  Operational 
Needs  Into  the  TES,  ICD,  and  TDS 

I  T.. ..  I.  "  " 


Initial  Capabilities  Document  (ICD)  Objectives 

(Defense  Acquisition  Guidebook  -  Section  9. 1.3.1  +  Other  Sources) 


Broad  Operational  Goals  and  Requisite  Mission  Capabilities  that 
drive  the  TES 


■  Addresses  specific  capability  gaps  in  terms  of 

□  Functional  areas 

□  Range  of  Military  Operations 

□  Desired  Effects  and  Time 

■  Description  of  1st  technology  demonstration 

Describes  materiel  and  non-material  approach  to  satisfy 
capability  gap 

■  Used  for  Milestone  A  decisions 


An  important  effort  to  define  Initial  Capabilities  is  to  perform  a  demonstration  of 
conceptual  designs  including  Command  &  Control  Display  Concepts  for  operational 
scenarios  -  such  as  those  done  by  the  CPoF  and  CSoF  projects. 
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Use  Cases  provide  a  good  Link  to 
Operational  Test  Validations 


UC  127  Analyze  the  Downloaded  Contact  Picture 
UC  57  Generate  an  External  Communication  Message 
UC  41  Import  a  Weapon  System  Configuration 


•  Use  Case  (UC)  Definition:  A  use 

case  is  a  single  [operational]  task , 
performed  by  the  end  user  of  a 
system ,  that  has  some  useful 
outcome. 


UC24  Generate  an  Alternate  Navigation  Path 
Plan 

Purpose  -  To  generate  an  alternative  by  identifying 
certain  attributes  of  the  baseline  plan  in 
accordance  with  the  defined  s^  of  navigatk 
path  rules 


P4l>  Jlffo 


Main  Flow 

1.  System  displaysjist^df" available  path  options 
User  chooses^5ath  options  to  review 
Systenr^isplays  path  plan  option  details 
User  identifies  &  changes  path  option  attributes 
System  displays  impact  assessment  of  change 
adjustment 

User  chooses  to  save  alternate  navigation  path 
plan 

System  saves  alternate  navigation  path  plan 


2. 

3. 

4. 

5. 

6. 

7. 


•Use  cases  (UC)  are  a  popular 
way  to  express  operational  & 
rstem  requirements 

•  A  UC  spans  between  the  user 
needs  andsystem  functionality 

•  The JUc  directly  states  the  user 
interition  and  system  response 

each  step  in  a  particular 
interaction. 


•  Mission  Analyses  result  in 
defining  tasks  in  an  operational 
scenario  that  needs  to  be 
completed. 

•  These  tasks  become  the  basis 
for  defining  use  cases 

•  A  system  design  that  satisfies  a 
UC  meets  an  operational  need. 


A  good  Test  Strategy  includes  a  Test  Plan  that  performs  Test  Cases  involving 
Uses  Cases  for  both  DT&E  and  OT&E. 
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Test  Architecture  incorporates  Methodology  Outputs 
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TES,  ICD,  and 
TDS,  etc. 


Operational  Realism 
in  Test  Architecture 
Components 


_ _  _ ^ - 

_ _  "I 

"  Test  Scope 

•  SOW  emphasis  on  specific  efforts  to  verify 
operational  requirements _ 

Test  Documentation 
•  Test  Strategy,  Plans,  Procedures,  & 
Reports  for  each  level^ _ 

"  _ _ ^  Test  Obiectives 

‘  —  - - ^  m  L  ■■■  ■ji -i  JL 

•  Clearly  define  overall  project  operational  test 
objectives 

•  Define  multiple  levels  of  testing:  Laboratory, 

-  Test  equipment 

•  Technology  in  test  environments 

•  Transportability  for  operational  field  tests 

LFjeld,  and  Operational___^^'' 

— — — 

Test  Environments 

"  - Test  Metrics 

•  DT&E  and  OT&E  environment  HW/SW 

•  Operational  metrics 

•  Laboratory  vs.  Field  metrics 

^ - - — - ^  Test  Operations 

v _ _ _  M in p 

•  Planning  and  conducting  DT&  E  (T&I)  and  OT&E 

•  Plans  for  incorporation  of  OT&E  early  in  cycle 

•  Field  testing  and  Customer  site  acceptance  tests 

1  CjL  rICIIICi  fdllClll 

•  Test  Manager  &  Test  Director 

•  Customer  Management  for  Field  Testing 

- 

-  - ~-'"1 
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Plan  Strategies  for  each  Level  of 
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Testing 


Integration 

Plannina 

— 

System  1 

Subsystem  1 

Unit  1 

Platform 

Unit  2 

NCO  =>  System  of  Systems 

System  N 

Subsystem  N 

Unit  N 

□  The  System  Integration  Planning  Must  Address  All  Levels 
of  Test  -  Operational  testing  done  at  each  level  Ju 

■  Results  in  Multiple  Test  Plans 

□  Minimize  risks  of  meeting  Operational  requirements 


•  Integrating  DT&E  Events  with  early  OT&E  Events 

•  DT&E  is  Laboratory  level  testing  -  little  or  no  human  environment 

•  DT&E  uses  simulators  for  operating  environment  conditions 

•  OT&E  includes  the  human  element  in  testing 

•  OT&E  has  the  real  platforms  and  real  operating  environments 

•  Early  OT&E  Alignment  in  DT&E  environment  can  be  done 

•  Plan  long  duration  operability  demonstration  tests  with  real  system  operators 

•  Schedule  regular  test  shifts  for  3-6  months  for  real  system  operators 
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□  Get  testable  operational  requirements  [e.g.  use  cases] 
defined  early  and  agreed  upon  with  the  Customer 

□  Getting  the  users  involved  early  and  often  results  in  you 
building  what  the  operational  users  want,  not  what  they 
asked  for 

■  Often  when  asked  to  clarify  a  requirements,  the  real  need  is 
uncovered... not  the  "design"  they  "required" 

■  Results  in  fleet  buy-in  and  more  likely  for  operational 
acceptance 

□  The  Navy  ARCI  project  has  CONOPS  groups  to  address 
capabilities  gaps  currently  not  being  supported  now. 

■  After  group  meets,  then  they  meet  with  contractors 

■  Initially  a  new  capability  could  be  requested  and 
implemented  without  broad  need.  The  group  solves  that. 

□  Often  involving  an  operational  crew  in  laboratory  testing 
will  identify  design  improvements  and  improve 
acceptance  later 
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1.  Incorporating  operational  realism  early  will  result  in 
building  something  that  be  used  and  verified  in  DT&E  and 
OT&E 

2.  A  methodology  exists  and  is  being  performed  by 
contractors  that  addresses  operational  realism  early  in 
the  design  process 

3.  Conducting  the  operational  modeling,  designs,  and 
technology  trade-offs  will  result  in  requirements, 
strategies,  and  technology  candidates  for  including  in  the 
TES,  TDS,  and  ICD. 


Systems  Engineering  provides  a  structured  approach  to  managing  the  technical  solution 
over  the  full  life  cycle  from  concept  to  deployment  to  retirement... 

...Test  and  Evaluation  complements  this  approach  with  support  for  defining  requirements 
and  integration  planning. ..and  conducting  many  levels  of  integration  tests  with  systems 
engineering  support  to  achieve  customer  acceptance  of  a  system... 
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Abstract 


Implementing  a  Methodology  to  Incorporate  Operational  Realism  in  CONOPS  &  Testing 

Session:  Test  and  Evaluation  in  Systems  Engineering 

Operational  realism,  a  key  piece  of  an  Operational  View  in  a  System  Architecture,  is  today  being  implemented  as  part  of  a  Double  Helix 
Methodology.  The  methodology,  developed,  tested,  and  validated  by  the  DARPA  Army  Command  Post  of  the  Future  is  being  used  by 
ASSETT  for  a  future  Navy  Combat  System  of  the  Future.  The  methodology  incorporates  iterations  of  CONOPS  and  Technology  trade-offs 
using  Subject  Matter  Experts  (SME),  web  exercises,  interviews,  team  experiments,  and  display  simulations  in  developing  and  testing  evolving 
conceptual  system  designs  prior  to  a  system  acquisition.  This  presentation  will  identify  how  ASSETT  Inc.  has  successfully  implemented  this 
approach  within  its  system  engineering  process  and  how  it  will  eventually  lead  to  better  acquisition  development  and  test  strategies. 

The  Double  Helix  Methodology  and  the  CONOPS/Technology  Trade-offs:  In  the  2008  NDIA  DTE  Committee  Study  Task  Report,  one  of  the 
key  findings/recommendation  was  “to  include  operational  realism  in  early  phases  of  acquisition  of  a  new  system  during  generation  of  the 
CONOPS,  ICD,  TDS,  and  TES”.  Our  Mission  Driven  Design  process  uses  the  Double  Helix  Methodology,  beginning  with  a  conceptual 
CONOPS  and  an  eye  for  the  future.  New  automated  capabilities  are  envisioned  based  on  a  decision  centered  design  approach  to  defining 
tasks,  their  sequence,  and  any  associated  time  constraints.  The  CONOPS  is  synchronized  iteratively  with  the  technology  team  to  address  the 
CONOPS  expectations  for  the  future  technologies  &  promising  capabilities.  From  mission  phases  in  operational  scenarios,  many  different 
uses  cases  can  be  defined  to  test  this  new  operational  realism  in  DT&E  and  OT&E. 

Outputs  of  the  Approach  Feed  System  Capabilities  and  Strategy  Documents:  The  new  capabilities,  technologies,  and  mission  driven 
conceptual  designs  will  derive  requirements  to  be  captured  in  the  acquisition  development  and  testing  documentation.  This  presentation  will 
provide  an  insight  into  the  methodology,  the  Decision  Centered  Design  process  that  drives  the  operational  and  system  architecture  views,  how 
the  decisions  are  used  in  defining  the  tasks  and  events  in  each  evolving  CONOPS/Technical  iteration,  and  how  the  Testing  &  Simulations  of 
the  HSI  displays  using  operational  personnel  will  focus  the  designs  for  a  system  to  be  acquired. 
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22  October  2008 


Slide  20 


Systems  Engineering  in  the  S&T 
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Best  Practices  and  Other  Lessons  Learned  from  the  Air 

Force  Research  Laboratory 
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Overview 

*  AFRL’s  SE  Problem 

*  The  TASE  Study 

*  TASE  Assessment  Results  -  Best  Practices 

*  TASE  Recommendations 

*  Conclusions 
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AFRL’s  SE  Problem 


*  Technology  development  and  maturation  are  a 
contributing  element  to  the  acquisition 
process 

*  Recent  acquisition  “failures”  have  resulted  in 
an  increased  DoD  focus  on  systems 
engineering 


*  AFRL  is  also  being  asked  to  do  more  with 
fewer  resources 


So  -  why  shouldn’t  AFRL  apply  systems 
engineering  in  its  activities? 
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AFRL’s  SE  Problem  -  Continued 


Because... 

—  “SE  is  acquisition  oriented,  and  we  do  research” 

—  “AFRL  programs  are  small  with  limited  budgets, 
and  SE  adds  a  resource  burden” 

—  “SE  focuses  on  customers  and  requirements 
satisfaction,  and  research  programs  don’t  have 
either” 

—  “Structured  approaches  like  systems  engineering 
will  stifle  creativity  in  research” 


“We  don’t  need  no  stinking  SE!” 
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■ 


*  AFRL  commissioned  the  Transformational 
Activities  in  Systems  Engineering  (TASE) 
study  in  2006 

•  3  Phases 

—  Assess  AFRL’s  current  SE  state  of  practice: 
determine  DoD/AF  requirements;  assess  current  SE 
policy,  practices,  and  tools  (2006) 

—  Recommend  improvements  to  AFRL’s  SE  policy 
and  practices  (2007) 

—  Implement  and  sustain  an  approved  AFRL  SE 
process  (2008+) 
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TASE  Assessment  Process 


Assessment  based  on: 

—  Review  of  DoD  and  AF  SE  guidance 

—  Interviews  with  AFRL  Advanced  Technology 
Demonstration  (ATD)  and  other  high-priority 
program  personnel  (52  programs  assessed) 

Facilitated  by  GD-AIS  contractor  team 

—  5  senior  systems  engineers 

—  Former  Director  of  the  AF  Center  for  Systems 
Engineering 
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TASE  Assessment  Results 


Intent  of  DoD  guidance  encourages  us 
in  research  activities 

SE  was  not  foreign  to  AFRL 
programs  used  a  full  set 


f  SE 


The  S&T  environm'flS  different” 

—  Variable  proTBSajprze 

—  “Soft”  '  ^ments  (aka  “desirements”) 

(vs  hierarchical)  relationships 

— Testability  in  customer  base 


,  but  few 
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AFRL  S&T  Systems  Engineering  Example: 
Requirements  Development  and  Roadmapping 


*  AFRL  use  of  the  Integrated  Product  and  Process 
Development  (IPPD)  process 

—  High  Energy  Laser  on  a  Large  Tactical  Platform  (HELLTP) 
—  Next  Generation  Unmanned  Aerial  System 
—  Multiple  small  programs 

*  SE  Successes 

—  Increased  understanding  of  “customer”  needs 
—  Better  focus  on  which  technology  areas  to  pursue 
—  Increased  potential  for  successful  transition 


GENERAL  DYNAMICS 
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AFRL  Systems  Engineering  Example: 
Full  Systems  Engineering  Implementation 


*  The  Advanced  Tactical  Directed  Energy  System  (ATADS)  ATD 
used  SE  processes  to  successfully  meet  its  program  objectives 

—  Result  was  up  to  an  order  of  magnitude  reduction  in  weight  and  cost 
from  the  existing  airborne  infrared  countermeasures  system  with 
increased  performance 

•  SE  Successes: 

—  Lab-led  requirements  development  and  management  including  IPT 
with  user,  PO,  and  contractor  resulted  in  responsive  but  controlled 
requirements  that  balanced  user  needs  with  technical  realities 

—  Continuous  risk  management  successfully  responded  to  technology 
and  program  issues 

—  Model-based  decision  analysis  improved  both  requirements  and 
design  choices 

—  Strong  contractor  SE  processes,  monitored  by  Lab  managers, 
ensured  matured  technologies  and  integration  met  Lab  needs 


GENERAL  DYNAMICS 
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AFRL  Science  &  Technology 
Systems  Engineering  Best  Practices 

*  Requirements  Development  and  Decision  Analysis 

—  Formal  IPPD  process  tailored  to  AFRL’s  environment  and 
“Standardized”  between  Directorates 

—  Strong  Integrated  Product  Teams  (IPTs) 

*  Risk  Management 

—  Continuous  process  involving  AFRL  and  contractor 

*  AFRL/Contractor  Relationship 

—  Strong  contractor  SE  with  AFRL  understanding  and  oversight 

*  Senior  Leadership  Support 

—  Designated  Chief  Engineers  and  SE  Branches 


GENERAL  DYNAMICS 
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AFRL  S&T  SE  Best  Practice 

IPPD  Process 


Scene  Visibility  for  Out-of-Cockpit  Emissive  Surfaces 


0.0  -I - - 1  - , - r-1 - , - , - , - 

1.9  2.1  2.3  2.5  2.8  3.0  3.2  3.4  3.6 

Ratio,  LT  Source  to  LT  Scene,  as  viewed  through  the  filter 


Customer 

Requirements 


Technology 

Alternatives 


$ 


f  T 
De 

echnology 

monstrati 

1 

>n  , 

Value  Analysis 


Transition  Focused: 

•  Measurement-based  methods 

•  Balanced  tech  trades/options 

•  Quantify  desirability  &  risk 
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IPPD  Revisited 


Phase  1 : 
Expand  the 
problem 
space 


r 


Solution 
Independent 
Analysis 


Solution  Dependent  Analysis 
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TASE  Recommendation: 
Attack  the  Problem  on  2  Fronts 


Cultural  Change: 

—  Build  upon  current  SE  Best  Practices  in  AFRL 

—  Implement  a  tailored,  consistent,  and  complete  SE  framework 
that  is  a  part  of  everyday  operations  (not  a  “burden”) 

—  Provide  training  on  fundamental  SE  practices  tailored  to  the 
research  environment 

—  Champion  the  S&T  SE  framework  and  supporting  organization 
at  the  highest  level  of  leadership 


GENERAL  DYNAMICS 

Advanced  Information  Systems 
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TASE  Recommendation: 
Attack  the  Problem  on  2  Fronts 


*  Cultural  Change  and 

*  Process  Improvement: 

—  Institute  strong  requirements  development  and  decision 
analysis  processes 

—  Employ  continuous  technical  management  processes 

—  Ensure  AFRL  technology  program  managers  understand  and 
have  visibility  into  contract  SE 

—  Reduce  program  risk: 

*  Foster  customer  intimacy,  recognizing  customer  changes  as  a 
key  factor  in  transition  risk 

*  Investigate  technology  alternatives  early  in  the  program 


GENERAL  DYNAMICS 
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Conclusions 


*  AFRL  has  discovered  that  Systems 
Engineering  is  a  good  idea  for  S&T  work 

*  AFRL  has  learned  that  implementing  SE 
processes  must  be  attacked  on  2  fronts: 
cultural  change  and  process  improvement 

*  AFRL  is  implementing  process  and  culture 
improvement  efforts  base  on  Best  Practices 


GENERAL  DYNAMICS 

Advanced  Information  Systems 
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Questions? 


AFRL  POCs: 

—  Dr.  Ken  Barker  (Deputy  Director  for  Program 
Management  and  Systems  Engineering) 
kenneth.barker@wpafb.af.mil 

—  Mr.  Bill  Nolte  (Assistant  to  Dr.  Barker  for  SE) 
william.nolte@wpafb.af.mil 

General  Dynamics  POC: 

—  Mr.  Bill  Doyle,  PMP  (TASE  Project  Lead) 
william.dovle@qd-ais.com  (719-641-3758) 


GENERAL  DYNAMICS 

Advanced  Information  Systems 
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Overview  of  presentation 


*  How  is  DoD  viewed  today? 

*  How  is  SOA  related  to  net-centricity? 

*  What  would  be  a  more  SOA-like  approach 
look  like? 

*  Conclusions 


♦  SPEC 
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How  Is  SOA  Related  to  Net-Centricity? 


*  DoD  defines  net-centricity  as  “the 
realization  of  a  networked  environment, 
including  infrastructure,  systems, 
processes,  and  people,  that  enables  a 
completely  different  approach  to 
warfighting  and  business  operations.”* 


See  DoD  Net-Centric 
Data  Strategy,  May  9, 
2003 


•  The  idea  is  to  use  the  global  network  to  provide 
warfighters  and  decision  makers  with  the 
information  they  need,  when  they  need  it,  in  a 
secure  environment 

*  Service-Oriented  Architectures  provides  a 
means  to  package  business  processes  as 
interoperable  services 


So  does 
the 

current 
model 
lend  itself 
easily  to 
SOA? 


•  Hence,  it  is  seen  as  a  means  to  implement  net- 
centricity 


♦  SPEC 
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Proposed  SOA  Business  Model  for  DoD 


Product  and 
Service  Providers 


eg-: 

•  Information/ 
Data 

•  Content 

•  Services 


■*&> 


$ 

% 


\  V 

' 


#  <f/ 
^ .  V 


Information  & 
Service 
Consumers 


All  Domains 
and  COIs 


c"K 


Read  Data  & 
Information  r 

DoD  Enterprise 
Services 

Value-Added 

“Resellers” 

(e.g.,CES) 

Intelligence  (Post) 

(rl UUcoo ) 

Market  Status, 


Exchange  Regulators  (Guidance) 

•  Domain  &  COI 
Establishment,  Monitoring 
and  Control 

•  Business  Rules 

•  Regulations 

•  Policies 


Transaction 
Support;  Decisions 


Information  & 
Service  Brokers 
(Process) 


e.g.: 

•  Intelligence 
Analysts 

•  Algorithm 
Developers 


e.g.: 

•  Decision  Makers 

•  Chain  of  Command 

•  Expediters 


TPPU  +  G 
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*  Net-centricity  can  provide  significant 
benefits  to  warfighters  and  decision 
makers 

*  SOA  can  provide  an  effective  means  for 
implementing  net-centric  principles 

*  By  adopting  a  business  model  that  cuts 
across  the  organizational  boundaries,  we 
can  reap  the  benefits  of  DoD’svision  for 
net-centricity 


©2008  Systems  and  Proposal  Engineering  Company.  All  Rights  Reserved 
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Proposal  Engineering  Company 


©2008  Systems  and  Proposal  Engineering  Company.  All  Rights  Reserved 


Overview  of  presentation 


•  Why  yet  another  “methodology?” 

•  What  is  KBAD? 

•What theory  underlies  KBAD? 

•  What  kind  of  tools  work  with  KBAD? 

•  What  process  does  KBAD  implement? 

•  What  kind  of  people  do  we  need  to  execute 
KBAD? 

•  How  do  we  move  from  drawing  pictures  to 
building  a  knowledgebase? 


♦  SPEC 
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Why  Yet  Another  Methodology? 


•We  have  the  DoD  Architecture  Framework  ... 

•  But  DoDAF  isn’t  a  methodology,  its  just  a  description 
of  necessary  products 

•We  have  UML  ... 

•  But  UML  is  only  a  software  engineering  technique. 

You  have  to  come  up  with  the  process  and  tools  for 
implementing  it 

•We  now  have  SysML  ... 

•  But  SySML  is  just  another  technique  and  still  needs 
more  definition  to  create  complete,  executable 
designs 

•What’s  missing? 

•  A  complete,  coherent  technique,  process,  and  tool  set 
that  results  in  a  knowledge  base  that  can  be  used  for 
full  lifecycle  decision  making 

♦spec 
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Knowledge-Based  Analysis  and  Design 


*  KBAD  combines  system  engineering  and 
program  management  disciplines  to  enable  the 
development  of  a  knowledgebase  that  can 
enable  cost-effective  decision  making 

*  KBAD  spans  the  acquisition  lifecycle  enabling 
support  for  design,  development,  integration, 
test,  operations  and  sustainment 

*  KBAD  focuses  on  using  a  variety  of  techniques 
and  tools,  brought  together  in  a  common 
database  using  special  software  to  migrate  data 
between  tools 


KBAD  reduces 
costs  and 
increases  speed 
of  delivery  by 
simplifying  the 
data  captured 
and  focusing  on 
the  analyses 
needed  for 
design.  The 
result:  a 

knowledge-base 
for  decision 
making. 


•  The  KBAD  process  links  the  technique  and 
tools  together  in  an  executable,  cost-effective 
way  to  support  decision  making  at  all  levels 


♦  SPEC 
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•Technique 

•  Modified  Model-Based  System  Engineering 
(MBSE) 


KBAD  was 
developed 
over  the  past 
15  years  and 


•  Process 


brings  lessons 
learned  from 


•  SPEC’s  Middle-Out  Process  for  Architecture 


those  years  of 


Development  and  System  Engineering 


experience. 


•Tools 


•  A  variety  of  COTS  tools  tailored  to  the  MBSE 
modifications  and  special  needs  of  DoDAF 

•  People 

•  Trained,  experienced  professionals  who  bring  a 
wealth  of  different  backgrounds  and  knowledge  in 
architecture,  system  engineering,  modeling  & 
simulation,  physics,  computer  science,  test& 
evaluation,  operations  &  support 

♦  SPEC 
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The  technique:  refined  MBSE 


*  Various  forms  of  model-based  system 
engineering  have  been  developed 

*  SPEC  uses  the  one  developed  by  TRW  in  the 
late  1960s,  which  has  been  successfully  used 
since  then 

*  SPEC  has  refined  this  technique  by 
simplifying  the  information  collected  (entities, 
relationships  and  attributes)  and  adding  a 
number  of  key  elements  missing  from  the 
original  development 


♦  SPEC 
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1.  Logical  architecture 

(behavior)  model 

•  Functional  sequencing 

•  Data  flow  and  size 

•  Resource  model 

•  Evolution  in  time 

2.  Physical  architecture 
(asset)  model 

•  Interface  definition 
(bandwidth  and  latency) 

•  Actions  allocated  to 
Assets 

•  Data  allocated  to 
interfaces 


Date: 

Friday,  February  08,  2008 

Author: 

Administrator 

Number: 

Name: 

MCVS  Subsystem 

♦  SPEC 
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Models  are  based  on  language 


Language 

Elements 

English 

Equivalent 

KBAD  Schema  Example 

Element 

Noun 

•  Statement 

•  Action 

•  Asset 

• 

Relationship 

Verb 

•  Statement  is  the  basis  of  an  Action 

•  An  Action  is  performed  bv  an  Asset 

• 

Attribute 

Adjective 

•  Description 

•  Type  (e.g.,  Operational  Activity  is  a  type  of 
Action 

• 

Attribute  of 
Relationship 

Adverb 

•  amount  of  Resource  consumed  bv  an  Action 

•  acquire  available  (hold  partial)  Resource  for 
Action 

• 

Structure  Enables 
Executability 

Graphics/ 

Drawings 

•  Graphic  Views:  Behavior,  Hierarchies, 
Physical  Block 

♦  SPEC 
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We  modified  Vitech’s  schema 


KBAD  CORE  Elements  Rationale 

Element 

Action 

Function/Operational 

Activity 

Provide  overall  class  for  actions 

Artifact 

Document 

Recognized  not  just  documents 

Asset 

Component/Operational 

Element 

Provide  overall  class  for  assets 

Characteristic 

type  of  Requirement 

Way  to  capture  metrics  and  other 
characteristics  of  an  element 

Cost 

attribute  of  Component 

Broadens  capture  of  costs 

Input/Output 

Item/Operational 

Information 

Clearer  name 

Issue 

Issue 

Same 

Link 

Link/Needline 

Provide  overall  class  for 
transmission 

Location 

none 

Captures  geolocation  information 

Risk 

Risk 

Same 

Statement 

type  of  Requirement 

Clearer  name 

Time 

attribute  of  Function 

Broadens  capture  of  times 

The  goal  was 
to  simplify  and 
clarify  the 
language. 


♦  SPEC 

_ _  lys'^rr^  rf-r 


©2008  Systems  and  Proposal  Engineering  Company.  All  Rights  Reserved 


We  related  all  the  KBAD  schema  elements 


Action 

Cost 

Characteristic 

Artifact 

Asset 

Input/Output 

Link 

Statement 

Issue 

Risk 

Time 

Location 

CORE 

Equivalent 

DoDAF  Equivalent 

Action 

decomposed  by 

incurs 

specified  by 

documented  by 

performed  by 
utilizes 

inputs 
outputs 
triggered  by 

- 

based  on 

generates 

resolves 

occurs 

located  at 

Function 

Operational  Activity/ 
System  Function 

Cost 

incurred  by 

decomposed  by 

specified  by 

documented  by 

incurred  by 

incurred  by 

incurred  by 

based  on 

generates 

incurred  by 

occurs 

located  at 

New 

N/A 

Characteristic 

specifies 

specifies 

decomposed  by 

documented  by 

specifies 

specifies 

specifies 

based  on 

generates 

causes 

occurs 

located  at 

New 

N/A 

Artifact 

documents 

documents 

documents 

decomposed  by 

documents 

documents 

documents 

source  of 

generates 

causes 

occurs 

located  at 

Document 

N/A 

Asset 

performs 
utilized  by 

incurs 

specified  by 

documented  by 

decomposed  by 

- 

connected  by 

based  on 

generates 

causes 

occurs 

located  at 

Component 

Operational  Node/ 
System  Node 

Input/Output 

input  to 
output  from 
triggers 

incurs 

specified  by 

documented  by 

- 

decomposed  by 

transferred  by 

based  on 

generates 

causes 

occurs 

located  at 

Item 

Operational 

Information/Data 

Link 

- 

incurs 

specified  by 

documented  by 

connects 

transfers 

decomposed  by 

based  on 

generates 

causes 

occurs 

located  at 

Link 

Needline/Interface 

Statement 

basis  of 

basis  of 

basis  of 

stated  in 

basis  of 

basis  of 

basis  of 

decomposed  by 

generates 

causes 

occurs 

located  at 

Requirement 

N/A 

Issue 

generated  by 

generated  by 

generated  by 

documented  by 

generated  by 

generated  by 

generated  by 

generated  by 

decomposed  by 

causes 

occurs 

located  at 

Issue 

N/A 

Risk 

caused  by 
resolved  by 

incurs 

caused  by 

documented  by 

caused  by 

caused  by 

caused  by 

caused  by 

caused  by 

decomposed  by 

occurs 

located  at 

Risk 

N/A 

Time 

occurred  by 

occurred  by 

occurred  by 

occurred  by 

occurred  by 

occurred  by 

occurred  by 

occurred  by 

occurred  by 

occurred  by 

decomposed  by 

located  at 

New 

N/A 

Location 

locates 

locates 

locates 

locates 

locates 

locates 

locates 

locates 

locates 

locates 

occurs 

decomposed  by 

New 

N/A 

Reduced  number  of  elements  from  21 

*  to  12,  while  adding  time,  location 

and  cost 

*CORE’s  DoDAF  schema 

♦  SPEC 
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A  key  attribute  -  type 


•  We  added  a  “type”  attribute  to  all  classes 


*  Each  “type”  attribute  contains  different 
designators  for  the  parent  class 

*  Examples: 

•  Assets  can  have  types  that  include: 

•  Operational  Node,  System,  Component,  Resource, 
Subsystem,  System  of  Systems,  Component,  ... 

•  Actions  can  have  types  that  include: 

•  Operational  Activity,  System  Function,  Task, 
Mission,  ... 


Using  the  type 
attribute  we 
reduce  the 
complexity  and 
ease  changes 
in  perspective 
from 

requirements 

to 

implementation. 


•  You  can  expand  these  lists  to  characterize 
anything  in  that  class 


•  When  we  display  the  element,  we  use  the 
type 


♦  SPEC 
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Benefits  of  the  KBAD  Schema 


*  Reducing  the  number  of  primary  data 


elements  means  less  complexity  for 
analysts  to  deal  with 


The  result  is  a 
more  cost- 


•  Less  complexity  enables  quicker  capture  and 
presentation  of  the  information  for  analysis  and 
decision  making 

*  Covers  programmatic,  as  well  as  technical, 
elements  of  information 


effective  means 
for  describing  an 
architecture  or 
system  design. 


•  Enables  the  trade  off  between  cost,  schedule  and 
performance  necessary  for  good  design  and 
decision  making 

•  Eliminates  overlap  between  similar  data 
elements 


•  Reduces  potential  for  duplication  of  information 
which  cuts  the  time  and  cost  of  data  gathering 

♦spec 

^  Systems  and 
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MBSE  Describes  Beha 


•Typical  data/activity 
modeling  only  works  in  the 
data  dimension  (e.g.  IDEFO 
or  Data  Flow  Diagrams) 

•  For  simple  systems  with 
sequential  flow,  this  is 
sufficient 

•  However,  for  more 
complex  systems,  which 
all  architecture  are,  it  can 
be  very  misleading 

•  We  need  to  be  able  to 
predict  how  system  will 
behave 


Functional  Sequencing 


The  missing  dimensions: 
resources,  physical  sizes  of 
data  and  interfaces 


'3"-Dimensions  of 
Behavior  Analysis 
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Why  is  sequencing  important? 


*  In  software  the  mantra  is:  data,  data,  data 


•  Why?  Because  a  tremendous  amount  of  software 
programming  has  to  do  with  input/output,  hence  the 
need  to  understand  the  data  very  well 

•  The  functional  sequencing  for  individual  software 
modules  is  relatively  simple  and  many  algorithms 
exist  for  complex  methods  (e.g.,  sorting  algorithms) 

*  In  architecture  development  (or  system 
engineering  or  business  process  modeling 
...)  sequencing  is  actually  more  important 
than  the  data 


Hence  the  real 
answer  is  we 
need  both  if  we 
are  to  develop 
systems  and 
services  with 
predictable 
behavior. 


•  We  want  to  know  how  the  data  affects  the 
functional  sequencing  -  we  call  these  triggers 

•  We  want  to  control  the  behavior  to  avoid  having 
significant  failures 

•  We  also  need  sequencing  for  the  human  side  p 
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MBSE  provide  a  robust  set  of  constructs 


SERIAL 


PARALLEL 


Exit  1 


Iteration  Set 

-0H. 

Action  A 

ITERATIVE 


SELECTIVE 


l 


DM 


(DM- 


LOOP 
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Exit  1 

1— ► 

Action  B 

(£ 

t 

Action  A 

Action  C 

Exit  2 

MULTIPLE-E 


Domain  Set 
with  coordination 


Action  B 


Action  A 


REPLICAT 
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One  diagram  gives  many  products 


-e 


EFFBD: 
complete  and 
executable 


O 


Text 


H  Monitor  Vehicle  Status  ssP-opertySheet  OTS. PROJECT  2-8-08) 

Me  Edit  View  Dito  loots  Help 

H  AMD  d  i  3*  a1 


I  The  three  process 


this  DFD  pto'.nde  the  Meritor  V  ehdr  Status 
facility  within  the  Provide  V  ehicle  Monitonnc  and  Control  function 

and  data  prodded  from  the  roadway  on  wfcach  the  vehde 


Thuradav.  Decemoer  27,  2D07M  11:12:33  AM 


Targets  *  Atatutn 


augwneob*  i_i 

based  on  It1 

Sort  |*ime>cbrdass  | 

RWDA  Last  Modlied  Thursday.  February  07  2008  at  OP  54  36  AM 

FFBD 

lacks 

data 


IDEFO: 

lacks 

constructs 


OV-5;  SV-4 


N2  Chart: 
lacks 

constructs 


emergency 
vehicle  priority 

intersection 

collision_ 

avoidance_data 


post  or  warnings 


safety  warnings 

driver_safety_ 


Thursday,  February  07,  2008 


Hierarchy:  only 
parent  to  child 


OV-6c; 

SV-IOc 


Monitor  Vehicle  Status 


Monitor  Vehicle  Status 


Operational  Activity 


OV-5;  SV-4 


1 

1  - 

- 1 

3.1.1 

3.1.2 

3.1.3 

Produce  Collision  and 

Carry-out  Safety 

Process  Vehicle 

Crash  Avoidance  Data 

Analysis 

On-board  Data 

System  Function 

System  Function 

System  Function 

Date: 

Thursday,  February  07,  2008 

Author 

Administrator 

Number: 

3.1 

Monitor  Vehicle  Status 

♦  SPEC 
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MBSE  also  diagrams  the  physical  elements 


Physical  Block 

Physical  Hierarchy  Diagram 
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Traceability  is  a  key  to  success 


Public  Roads 
J  uly/August  2007 


White  Paper 


(documents 

A 


Automated  ITS 
Project 


Program 


|document7 


PR.l 

Need  for  protected, 
dedicated  lanes? 


|document7 


PR. 2 

Driver  acceptance? 


documents 


PR. 3 

Vehicle  and  highway 
systems  that  operate 
at  a  higher  level  of 
reliability  and  perfo... 


Increased  liability  for 
manufacturers  and 
owner/  operators  of 
automated  systems? 


Pf 

S 


(allocated  to 

causes 

|causes 

results  in 

results  in 

results  in 

results  in 

S.l 

S.6 

S.14 

Automated  Intelligent 
Transportation 
System  Context 

Loss  of  Privacy 

Potential  Resistance 
by  Organizations 

Single  car  enters 
roadway 

Single  car  traveling 
sudden  obstacle 

Worst  case  scenario 

Proposed  Liability 

Leg  is  lation 

Architecture 

Other 

Other 

Operational  Activity 

Operational  Activity 

Operational  Activity 

Policy 

♦  SPEC 
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Tools  support  the  technique  and  process 


Most  Programs 
Require  Tools  in 
All  These 
Domains,  but ... 


CORE® 


Requirements 
Analysis  Tools 

(e.g.  DOORS) 


netViz 


Functional 
Analysis  Tools 

(e.g.  System  Architect) 


Program 
Management  Tools 

(e.g.  MS  Project) 


Hardware  Design 
Tools  (e.g.  CAD) 


Operations  & 

Support  Tools 
(e.g.  BPEL) 


Test  &  Evaluation 
Tools  (e.g.  M&S) 


Software  Design 
Tools 

(e.g.  Rational  Suite) 


...they  do  not 
interoperate 
well  together. 


MD  Workbench 


SPEC’s  KBAD 
methodology 
uses  CORE  and 
MD  Workbench 
to  provide  the 
underlying  tool 
interoperability. 


♦  SPEC 
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Tools  used:  CORE 


p  o  r  a  t  i  o  n 


that  provides  control 
of  the  TUAV  by  the  JTF 


Ref. 


CORE'S  system  engineering 
tools  maintain  an  integrated 
design  repository  that 
provides  traceability 
between  requirements, 
functional  models  and 
system  design  elements 

CORE'S  database  schema 
may  be  modified  to 
customize  the  tool  to 
support  customer  needs 
and  facilitate  tool 
integration 

Executable  diagrams 

Special  schemas  and 
reports 

Powerful  scripting  language 
for  your  own  report 
generation  Version  5.1  released  with 

updated  schema  and 
reports 


S.1.1 

S.1.2 

Platoon  ^ 

Receive  and  Act  on 
Information 

This  represents  one 
option  for  the  system 

► 

Call  for  Information 

r 

Verbal  RFI 


Verbal 

Information 


Brigade 


S.1.3 


Receive  Request 
for  Information 


S.1,4 


Request  Latest 
Information  for 
Location 


S.1.5 


Translate 
Information  into 
Verbal  Commands 


Formatted  RFI 


Collected 

Information 

P 

JTF 


S.1.6 


Receive  Formatted 
RFI 


S.1.7 


Task  Collection 
Management 
System 


S.1.8 


Transmit  Collected 
Information 


www.vitechcorp.com 

♦  SPEC 


Ref. 


©2008  Systems  and  Proposal  Engineering  Company.  All  Rights  Reserved 


Tools  used:  MD  Wor 


Eclipse-based  IDE  for  code 
generation  and  model 
transformation,  devoted  to 
implementing  MDA/MDE 
strategies.  It  provides: 

•  code  generation  (via  text  template 
engine  and  optionally  Java) 

•  model  manipulation  through 
dedicated  languages 

•  (imperative  rules,  declarative  ATL 
modules  to  support  QVT 
transformations,  Java) 

•  model  and  metamodel 
management,  including  UML 
support 

•  customizable  model  connectors 
(XMI  1.0  to  2.1,  XML,  Hibernate, 
COM,  etc.) 


I 


2.1  is  a  major  release!  New  features  of  MDWorkbench  2.1  includes: 


en< 


—^Microsoft  Word®  Document  generation 


MDWorkbench  2.1 

r^dgwrilpad.l 


Doc  Templates  enable  you  to  easily  generate  Microsoft  Word®  documents  (Word  2003  or 
2007). 

A  doc  template  is  edited  in  Word,  using  all  the  functions,  formatting,  and  styles  available, 
with  the  addition  of  MDWorkbench  constructs  to  access  model  data.  A  doc  template  brings 
the  power  and  ease  of  use  of  text  templates  into  Microsoft  Word®. 


»  Document  generation 
»  RCP/Command  line 
»  Navigation  history 
»  On -demand  reload 
»  Multiline  text  dialog 
»  UML  2. 1  upgrade 
»  Improved  reader 
»  Rose  model  reader 


You  can  generate  rich  documents  to  describe  models,  to  report  metrics,  to  report  QA 
validation  results,  etc. . .  _ 


http://www.mdworkbench.com 


A  great  way 
to  move  data 
between 
different 
tools. 


♦  SPEC 
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Tools  used:  NetViz 


•  Personnel  all  over  the  world  use 
netViz  to  graphically  depict 
operational  architectures  and 
logistical  scenarios 

•  With  NetViz  you  can  create  the 
SV-1  and  SV-2  diagrams,  with  its 
intuitive  graphical  workspace, 
drill  down  capability,  and 
connectivity  views 

•  You  can  use  the  data  embedded 
in  your  netViz  projects  to  create 
other  critical  elements  of  a 
comprehensive  C4I 
documentation  project,  like  OV- 
1s  (Operational  Concept 
Diagrams)  and  OV-3s 
(Information  Exchange  Matrices) 


Version  7.1  released;  Available  in  Client 
Server  or  Enterprise  Web  editions  as  well 


V  SPEC 


www.netviz.com 


netViz 
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SPEC  processes  -  full  lifecycle 


Propoael  Engineering  Company 
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SPEC’s  middle-out  process 


Requirements 
Analysis 


Ol, 


Best  Use: 
"Classical  SE" 


Functional 
Analysis  and 
Allocation 


Best  Use: 
Architecture 
Development 


(To-Be) 


Adapted  from  E/A-632 


% 


Best  Use:  Reverse 
Engineering  (As-is) 


System 
Analysis  and 
Control 


Synthesis 
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Middle-out  timeline  with  products 


Requirements  Analysis 
Functional  Analysis 


5.  Develop  the  Operational  Context  Diap- — 

-OV-l-OV-7 


_ |  Synthesis 


6.  Develop  Operational  Scenarios 


OV-2 1 


7.  Derive  Functional  Behavior 


OV-3 


System  Analysis 
and  Control 


3. 


OV-6.  SV- 10  T 


1:2;i Perfpmi ipynamid  Analysis;  5^.7 


11.  Define  Resources,  Error  Detection  &  Recovery 


Conduct:  trade-off  Analyses 


16.  Generate  Operational  and  System  Architecture  Graphics,  Briefings  and  Reports 


Time 


♦  SPEC 


The  middle-out 
approach  has 
been  proven  on 
a  variety  of 
projects. 
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People  Considerations 


*  Large  teams  make  organization  and  focus 
on  a  vision  very  difficult 

*  You  need  people  with  a  wide  variety  of 
skills  and  personalities 

•  Someone  with  vision 

•  Someone  who  can  perform  the  detailed  system 
engineering 

•  Someone  who  understands  the  domain 

•  Someone  familiar  with  the  technique  and  tools 

•  Someone  who  understands  the  process 

*  They  need  to  be  trained  as  a  team  - 
including  the  government  personnel 


4  SPEC 
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rHow  Do  WtTMove  from  Drawing  Pictures  to 
JiujJding^aJCnowJedgebase? _ 


•  Apply  a  proven,  model-based  technique 
that  results  in  executable  diagrams 

•  Use  a  process  that  implements  the 
technique 

•  Use  industrial-strength  system 
engineering  tools 

•  Make  sure  the  personnel  who  use  the 
methodology  have  the  proper  knowledge, 
skills  and  abilities  to  implement  the 
approach 


♦spec 
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Questions  &  Discussion 


Integrated  Change  Control  for  the 
Concurrently  Developed  Complex 
Systems  -  Lessons  Learned 

Alexander  J.  Polack 
The  Aerospace  Corporation 


22  October  2008 


©  The  Aerospace  Corporation  2008 


Before  We  Begin... 


There  were  many  contributors  to  this  effort. 
Thank  you  everyone  who  helped! 


2 


Advanced  Extremely  High  Frequency  (AEHF)  System 


Reprinted  courtesy  of  the  United  States  Air  Force  Reprinted  courtesy  of  the  United  States  Air  Force 


•Mission  -  Provide  protected  satellite  communications  for  strategic  and 
tactical  defense  missions 

•Designed  to  augment  and  eventually  replace  the  Milstar  system 

•AEHF  Program  Office  is  located  at  the  Space  Missile  Center  (SMC), 
Los  Angeles  Air  Force  Base 


AEHF  Program  Challenges 


•  Concurrent  development  and  acquisition  of  major  AEHF  system 
elements 

•  Concurrent  development  of  interfaces 

•  Most  elements  have  different 

-  Contracts 

-  Contracting  agencies 

-  Contract  schedules 

-  Development  teams 

•  Backward  compatibility  requirements  with  existing  operational  systems 

•  Operational  systems  are  in  the  process  of  changing  while  in 
sustainment  mode 

•  New,  post  contract  award  requirements 

•  International  Partners 

•  Budgetary  and  regulatory  requirements  and  constraints 
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Change  Dilemma... 


•  Change  is  inevitable  on  a  large,  multi-year,  concurrent  development 
program 

•  Change  is  disruptive  by  its  nature 

•  Managing  change  is  not  easy 

•  Having  a  well  defined  and  understood  process  for  managing  change  is 
imperative 

•  Processes  need  to  be  constantly  adjusted  to  reflect  the  needs  at  hand 
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In  the  Beginning... 


•  AEHF  Program  Office  Change  Process  existed  since  the  beginning  of 
the  program 

•  December  2003  -  SMC/CMMI  Program  Office  Assessment 
recommends  review  of  the  existing  change  process 

•  September  2004  -  Comprehensive  review  of  the  AEHF  Change 
Process  is  initiated 

•  July  2005  -  “New  and  Improved”  AEHF  Change  Process  makes  its 
debut 
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What  We’ve  Learned  About  the  AEHF  Change  Process  Since... 


•  Define,  document,  and  implement  the  process 

-  Identify  what  needs  to  be  accomplished,  e.g.,  Engr.  Change  vs.  Contr.  Change 

-  Know  your  stakeholders 

-  Provide  enough  detail  to  map  it  into  the  process  above  (e.g.  Group  to  Wing) 

-  Define  Entry  and  Exit  criteria  for  each  step 

-  Identify  Artifacts  created  and  modified 

-  Define  realistic,  nominal  timelines 

-  Apply  a  “KISS”  principle  at  every  opportunity 

•  Train,  train,  and  train  again 

•  Execute  and  measure  process  performance 

•  Implement  Process  Volume  controls 

-  Addresses  multiple,  simultaneous  changes  and  resource  contention 

•  Adjust  the  process  as  needed 

-  Conduct  process  improvement  activity  (e.g.,  VSM) 

-  Implement  changes  as  needed  and  as  possible 

-  Avoid  “Big  Bang”  approach  to  changes,  “evolutionary”  vs.  “revolutionary” 

•  Be  vigilant  about  your  process 
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AFS021  VSM 

AEHF  Change  Process  Current  State  -  July  2006 
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□ 

-  Pre-RFP 

□ 

-  Proposal 

□ 

-  Post  Proposal 

□ 

-  Contract  Mod 

•  1  -  Pre-RFP 

•  2  -  Proposal 

•  3  -  Post  Proposal 

•  4  -  Contract  Mod 


AFS021  VSM  Process  Activity  Value  Definitions 


Pure  Value  Activities 

•  Activities  that  change  the  form,  fit  or  function  of  the 
product/service  and 

•Activities  that,  when  asked,  the  customer  is  willing  to  pay  for  and 
•Activities  done  right  the  first  time. 

Business  Value  Activities 

•  Activities  causing  no  value  to  be  created  but  that  cannot  be 
eliminated  based  on  current  state  of  technology  or  thinking 

•  Required  (regulatory,  customer  mandate,  legal) 

•  Necessary  (due  to  non-robustness  of  process,  currently  required) 

Non  Value  Activities 

•  Activities  that  consume  resources  but  create  no  value  in  the  eyes 
of  the  customer 

•  Pure  waste 

•  If  you  can’t  get  rid  of  the  activity,  it  turns  to  yellow. 


AFS021  AEHF  CP  Initial  State  VSM  Analysis 


BCB1 
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MCB-1b 


TOTAL 

Task 

Trigger 

Done 

Cycle  Time  (days) 

321 

Touch  Time  (days) 

40.25 

TAKT  Time 

No.  of  People 

492 

Items  in  In-Box 

No.  of  Approvals 

143 

Distance  Item  Travels 

ESH  Issue 

%  Rework 

Top  3  Rework  Issues 

25  STEPS 

10  GREENS  40% 


7  REDS 


28% 


Wait  Time  (%)  87% 


AFS021  VSM  AEHF  Change  Process  Future  State  -  July  2006 
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AFS021  Value  Stream  Mapping  Event  -  July  2006 
AEHF  Change  Process 


Reprinted  courtesy  of  the  United  States  Air  Force 


Reprinted  courtesy  of  the  United  States  Air  Force 


What  Existed 

-  25  Steps,  321  Days 

of  cycle  time  (Excluding  Mod  Phase) 

What  We  Did 

-  Eliminated  steps  -  Consolidated  board 
meetings 

-  Optimized  Process  Flow 

•  Performing  technical  and 
programmatic  coordination  in 
parallel 

•  Improved  Organizational  Impact 
Analysis 

-  Started  Activities  Earlier  -  Improved 
Shoulder-to-Shoulder  (StS)  process  to 
allow  the  Tech  Evaluation  to  begin 
during  the  Proposal  preparation  phase 

Results 

-  Excluding  Mod  Phase 

-19  steps  -  24%  Improvement 

-  Cycle  time  196  days  -  39% 
Improvement 
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Number  of  Changes 


Metrics  -  How  Are  We  Doing? 

Start  through  Contract  Modification 


AEHF  Change  Process  Timelines  (233) 
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Bl  May-07 

□  Oct-06 

□  Mar-06 
■  Nov-05 


•  17  ECPs/CCPs  put  on 
contract  (05/05-  10/05) 

•  23  ECPs/CCPs  put  on 
contract  (05/05  -  03/06) 

•  44  ECPs/CCPs  put  on 
contract  (05/05  -  1 0/06) 

•  58  ECPs/CCPs  put  on 
contract  (05/05  -  05/07) 


•  Median 

-  1 1/05  -  303  days  ~  43  weeks 

-  03/06  -  252  days  ~  36  weeks 

-  10/06  -  243  days  ~  35  weeks 

-  05/07  -  233  days  ~  33  weeks 


30%  Improvement  including 
Mod  Phase 


VSM  Lessons  Learned 


•  VSM  technique  is  a  valuable  tool  in  identifying  “waste”  in  a  process 

•  Keep  the  team  lean  and  effective  -  10-15  people 

•  Must  have  representation  from  all  stakeholders 

•  Participants  need  to  know  the  current  process 

•  Participants  need  to  have  basic  training  in  process  improvement  techniques 

•  Need  experienced  event  facilitators 

•  Do  not  allow  changes  in  team  membership  once  the  event  starts 

•  Team  leaders  need  to  stay  engaged  throughout  the  event,  especially  during 
the  “heavy  lifting”  activities 

•  Team  leaders  must  be  careful  not  to  dominate  the  discussion 

•  Team  leaders  must  make  sure  the  discussion  does  not  deviate  to  far  from  the 
plans 

•  Be  vigilant  to  keep  the  “out-of-bounds”  items  out  of  discussions 

•  Have  fun! 
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CMMI  AEHF  Program  Office  Assessment  2007  -  Excerpts 


•  SMC  Tailored  CMMI® /Acquisition  models,  no  numerical  rating  or 
process  quality 

•  AEHF  Best  Practices  -  Within  the  Model 

-  A  rigorous  Change  Management  process  is  used  to  baseline  and  maintain 
requirements 

-  All  types  of  program  changes  are  analyzed  via  the  Change  Management 
process 

-  A  rigorous  Change  Management  System  of  boards  and  reviews  includes 
the  relevant  stakeholders 

•  Strengths  Above  the  Model 

-  Baseline  ECO  Board  (BEB)  Master  Matrix  and  waterfall  chart  are  used  to 
regulate  change  management  process  flow 
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Summary 


•  A  comprehensive  Change  Process  has  successfully  supported  the 
AEHF  program  for  the  past  3  years 

•  Further  improvements  are  possible,  necessary,  and  are  being 
implemented 


Any  Questions? 


All  trademarks,  service  marks,  and  trade  names  are  the  property  of  their  respective  owners 
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/^DEF  I  N  I  N  G  THE  FUTURE 


Rapid  Force  Structure  Analysis 

Capability  Effectiveness  Tool 

October  22,  2008 


David  Blancett 

Mgr,  Systems  Analysis  &  Simulation 

Kurt  Dittmer 

Dir,  Advanced  CONOPS 
Northrop  Grumman  Corporation 


Overview 


NURTHROR  GRUMMAN 


•  Large  trade  spaces  limit  M&S  effectiveness  for  force  structure  architecture  studies 

-  Problem  of  interest  was  Layered  I  ntelligence,  Surveillance  and  Reconnaissance  (LI  SR) 
with  I  ntegrated  Air  and  Space 

-  Solution  spaces  ranged  from  10,000  to  450,000+  possible  architectures 

•  Historically,  addressed  with  a  combination  of  common  sense  and  expert  opinion 

-  This  in  no  way  guaranteed  most  cost  effective  solution  was  truly  identified 

•  Three-part  structured,  traceable  process  was  developed  to  address  this  limitation 

-  Capture  commander's  intent  for  a  given  operation  and  translate  into  collection 
requirements 

-  Assess  force  structure  effectiveness  as  the  percentage  of  collection  requirements  met 
and  calculate  wartime  and  life-cycle  costs 

-  I  dentify  highest  potential  architectures  and  key  elements 

•  Three  tools  used  to  support  the  upfront  process: 

-  Collaborative  Reasoning  Tool  (CRT) 

-  Capability  Effectiveness  Tool  (CET) 

-  Analyst's  Workbench 


Process  Goal:  Identify  Limited  Set  of  Architectures  with  the 
Highest  Cost  Effectiveness  Potential  As  Starting  Point 

for  Detailed  Studies 
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Layered  I  SR  Problem  Definition 


NORTHROP  GRUMMAN 


Collection  capabilities  change 
with  each  scenario  and  phase 
of  war 

Joint  air,  space,  maritime  and 
ground  ISR  assets  have 
varied  and  overlapping 
capabilities 

National  leadership  must 
integrate  Irregular, 
Catastrophic  and  National 
Collection  capabilities  to 
make  informed  deployment 
decisions 


Group  Force  Mixes  by  Effectiveness/Cost 


Single  AOI 
Analysis 


Mission 

Analysis 


Common 
Force  Mix 
Reduced 
Trade  Space 


National  Co 


Theater  X 


Other  24/7  lnt€ 
Requirements 


Catastrop 
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Where  We  Started 


NURTHROR  GRUMMAN 


•  I  nvestigate  Surveillance  Capabilities  Within  10  Year  Horizon  That  Could  Provide 
Ubiquitous,  Near  Real  Time,  Theater- Wide  Coverage 

-  Consider  Manned,  Unmanned,  Satellites,  Ships,  Ground  Based  Systems 

-  Multi-Spectral 

•  What  Force  Mix  Provides  Most  Cost  Effective  Means  of  Accomplishing  Goals? 
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Developed  Toolset  to  Address  Solution 
Space  Size 


NURTHRUR  GRUMMAN 


•  Collaborative  Reasoning  Tool 

-  Means  to  easily  capture  commander's  intent 
for  each  phase  of  an  operation 

-  Single  J  oint  Forces  Commander  (J  FC),  or 
consensus  of  a  group 

-  Distributed  Capability 

•  Capability  Effectiveness  Tool 

-  Assess  alternative  force  structure  options  in 
terms  of  potential  effectiveness  and  cost 

-  Graphical  User  I  nterface  (GUI ) 

•  Analyst  Workbench 

-  Means  to  quickly  review  and  understand  a 
large  database 


Filters,  tagging  and  other  tools  to  support 
data  analysis 


Capability  Effectiveness  Tool 


Collaborative  Reasoning  Tool 


NURTHROR  GRUMMAN 


CET  Model 


NURTHROR  GRUMMAN 


•  Assesses  the  ability  of 
alternate  force  structures 
to  achieve  the 
commander's  intent 

•  Static  assessment  of 
potential  capability  over 
12  hr  or  24  hr  period 
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Expanded  CET  Applications 


ISiOKTHROR  CRUMMJXM 


•  CET  has  been  adopted  by  USSTRATCOM 

•  Other  CET  applications  under  consideration: 

Dl  A/  STRATCOM  J  oint  Functional  Component  Command  I  SR  - 
Assessment  Tool  for  Theater  Apportionment 

NRO:  I  ntegrating  CET  with  Northrop  Grumman 
Corporation  I  SR  Test  Bed  I  ncorporate  National  I  ntel 
Priority  Framework 


J  -2  and  OSD:  Capabilities  Assessment  Tool  for  Battlespace 
Awareness  Functional  Capabilities  Board  Program 
Objective  Memorandum  Decisions 


USSTRATCOM:  I  SR  Global  Force  Management/  Global 
Force  Posture 
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Summary 


ISiOKTHROR  CRUMMJXM 


Means  to  rapidly  identify  high 
potential  solutions  within  a  large 
problem  set 

Provides  understanding  of  the 
contribution  of  each  potential 
element 

Provides  understanding  of  the 
capability  gaps 

Effective  use  of  analyst  time  and 
simulation  resources 


Structured,  traceable  process  providing  the  means  to  support 

LI  SR  force  structure  architecture  studies: 

Stand-alone  and  as  a  lead-in  to  detailed  work 
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Presentation  Objectives 


•  Introduce  a  research  platform  to  address  concurrent 
engineering  concerns  of  software-intensive  system 
development 

•  Propose  new  metrics  to  characterize  increment  coupling  and 
cohesion  in  complex,  aggregate  life  cycle  models 
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®  ULCM  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  The  Aerospace  Corporatio 
SM  Unified  Life  Cycle  Modeling  is  a  Service  Mark  of  The  Aerospace  Corporation 


Wisdom 


“To  understand  a  subject,  one  must  tear  it  apart  and 
reconstruct  it  in  a  form  intellectually  satisfying  to 
oneself,  and  that  (in  the  view  of  the  differences  between 
individual  minds)  is  likely  to  be  different  from  the 
original  form.  This  new  synthesis  is  of  course  not  an 
individual  effort;  it  is  the  result  of  much  reading  and  of 
countless  informal  discussions,  but  for  it  one  must  in 
the  end  take  individual  responsibility.  “ 


Quote  from  J.L.  Synge ,  “Relativity:  The  Special  Theory”  (1956),  p.  vii 


Introduction 


•  The  National  Security  Space  Defense  Acquisition  Challenge 

-  Chronic  cost/schedule  overruns  in  space  acquisitions 

-  Difficulty  with  validating  the  contractors’  plans 

-  Difficulty  with  implementing  proper  controls 

-  Difficulty  with  successfully  executing  Evolutionary  Acquisition  and  Spiral 
Development-related  policies 

•  One  of  the  Most  Significant  Root-Causes  Identified 

-  Concurrent  Engineering  is  pursued  without  proper  models  and  tools  to 
manage  concurrent  process  streams 

•  Proposed  solutions  involve  the  use  of  ULCM®  (Unified  Life  Cycle 
ModelingSM)  and  DSM  (Design  Structure  Matrix) 

-  ULCM®  is  an  Aerospace-developed  research  framework  and 
methodology 

-  DSM  is  a  widely  used,  visual  system  representation  tool 


ULCM®  -  The  64  Thousand  Mile  View 


•  ULCM®  is  an  intuitive,  pattern-based  approach  for  specifying, 
constructing,  visualizing  and  documenting  the  life  cycle 
processes  of  software-intensive  system  development 

•  ULCM®  is  aspiring  to  become  the  “Occam’s  Razor”  of  Life 
Cycle  Modeling 

-  The  medieval  rule  of  parsimony:  “Plurality  shouldn’t  be  assumed 
without  necessity” 

•  William  of  Ockham,  14th  century  philosopher 

-  The  Life  Cycle  Modeling  (LCM)  rule  of  parsimony:  All  life  cycle 
models  are  constructs  or  derivatives  of  a  small  number  of  basic 
life  cycle  modeling  patterns 

•  ULCM®  is  also  a  research  platform 

-  It  provides  a  foundation  for  a  consistent  and  universal  system 
development  methodology 


The  First  Principles  of  Unified  Life  Cycle  Modeling* 


1 .  Covered  process  domains  are  acquisition  and 
development  of  software-intensive  systems 

2.  The  fundamental  building  block  of  life  cycle  models  is  an 
increment 

3.  All  life  cycle  models  are  constructs  or  derivatives  of  a 
small  number  of  basic  LCM  patterns 

4.  LCM  is  synergistic  with  architecture,  architectural  concepts 
and  architecture  modeling 

5.  Proper  representation  of  life  cycle  models  requires 
multiple  views 

6.  Concurrent  processes  are  synchronized  via  anchor  points 


*  Source:  [Hantos  2007] 


Principles  #1 ,  #2,  and  #3 


•  Principle  #1:  Covered  process  domains  are  acquisition  and 
development  of  software-intensive  systems 

-  ULCM®  might  be  applicable  in  other  domains  as  well,  but  such 
use  was  neither  pursued  nor  verified 

•  Principle  #2:  The  fundamental  building  block  of  life  cycle 
models  is  an  increment 

-  Increment  is  a  conceptual  term,  refers  to  the  difference  between 
two  subsequent  releases  of  the  product 

•  Delivering  any  useful  functionality  requires  the  creation  of  at 
least  one  increment  of  a  system 

•  Principle  #3  :  All  Life  Cycle  Models  are  constructs  or 
derivatives  of  a  small  number  of  basic  LCM  patterns 

-  Since  the  fundamental  building  block  is  an  increment,  the 
ULCM®  definition  of  all  LCM  patterns  must  address  their 
relationship  to  the  creation  and  sequencing  of  increments 


Principles  #4  and  #5 

•  Principle  #4:  Life  cycle  modeling  is  synergistic  with  architecture, 
architectural  concepts,  and  architecture  modeling 

-  Product  Architects  answer  the  “What”  question 

-  Process  Architects/Project  Managers  answer  the  “How”  question 

-  However ,  both  activities  are  concurrently  iterated  during  the  life  cycle 

•  Principle  #5:  Proper  representation  of  life  cycle  models  requires 
multiple  views 

-  Based  on  related  experience  with  architecture  modeling,  it  is  clear  that 
having  multiple  views  is  always  necessary  when  modeling  complex 
entities 

-  The  question  is  how  many  is  necessary  and  sufficient? 

•  Currently  ULCM®  assumes  two  views  of  any  life  cycle  model 

-  However ;  only  one  of  them,  the  Enactment  View,  will  suffice 
to  demonstrate  concerns  related  to  increment  coupling  and 
cohesion 
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Principle  #6 


Principle  #6:  Concurrent  processes  are  synchronized  via  Anchor  Points 
-  What  are  Anchor  Points  (APs)  ? 

•  Intermediate  milestones  with  specific,  focused  objectives 


Total  Work:  Release  (Delivery)  Increment 


•  •  • 


Work  Bucket”:  Iteration  Increment 


-  The  idea  behind  Anchor  Points 

•  “Extreme”  Planning  and  Monitoring  &  Control  Approaches 

-  Ad-hoc,  “code-and-fix”:  Planning  horizon  is  the  next  iteration 

-  Waterfall:  Planning  horizon  is  the  end  of  the  Incmment 

•  “Stop,  Stabilize,  and  Regroup”  Approach 

-  Iterative  with  APs:  Planning  horizon  is  the  next  Anchor  Point 


ULCM®  Enactment  View  of  an  Increment 


Phase! 

Phase2 

Phase3 

Phase4 

Phase5 

Phase6 

< - Increment - * 

Legend: 

MR  -  Increment  Inception  Readiness 
ILO  -  Increment  Life  Cycle  Objectives 
ILA  -  Increment  Life  Cycle  Architecture 
IOR  -  Increment  Operational  Readiness 
IDR  -  Increment  Delivery  Readiness 
IED  -  Increment  End-Of-Life  Decision 
IEL  -  Increment  End-Of-Life 


In  ULCM®,  life  cycle  phases  of  an 
increment  are  intentionally  not  named 


-  Specifying  both  phase  content  and  anchor 
points  is  redundant 

-  Phase  content  stays  flexible;  phase 
activities  are  not  pre-determined 

-  Focus  is  on  achieving  anchor  point 

objectives  A 


Product-related  AP  Objectives  During  Development 

•  MR  -  Increment  Inception  Readiness 

-  Its  sole  purpose  is  to  mark  the  beginning  of  an  increment 

•  ILO  -  Increment  Life  Cycle  Objectives 

-  Definition  of  operational  concept,  scope,  and  top-level  requirements 

-  Architectural  and  design  options 

•  ILA  -  Increment  Life  Cycle  Architecture 

-  Refinement  of  operational  concept,  scope,  and  top-level  requirements 

-  Resolution  of  ILO  option-explorations,  commitment  to  a  feasible 
architecture  and  technology  solutions 

•  IOR  -  Increment  Operational  Readiness 

-  Operation  and  quality  is  demonstrated  in  development  environment 

•  IDR-  Increment  Delivery  Readiness 

-  The  work  product  created  in  this  phase  is  ready  for 

•  Delivery  to  the  end-user/customer,  or 

•  Higher-level  integration  and  test 


Challenges  of  Concurrent  Engineering 


•  The  usual  HW/SW  dialog 

-  Traditional  SW  Position:  Give  me  the  working  hardware ,  and  leave  me  alone! 

-  Traditional  HW  Position:  Here  are  the  specs,  see  you  at  final  integration.  Now 
leave  me  alone! 

-  What  Really  Takes  Place:  HW  is  frequently  changing  during  design.  SW  people 
are  frustrated  and  inefficient.  SW  always  ends  up  being  the  bottleneck 

•  Similar  situation  in  case  of  concurrently  developed  software  components 

•  Challenges,  challenges  ... 

-  The  Project  Manager’s  Challenge: 

•  Managing  (estimating,  planning,  monitoring,  and  controlling)  concurrent 
engineering  processes 

-  The  Process  Architect’s  Challenge: 


•  Dealing  with  life  cycle  modeling  complexity 

-  Concurrent  engineering  of  hardware  and  sol 

-  Iterative/incremental  processes 
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Anchoring  Concurrent  Engineering  Processes  in  ULCM® 


Hardware-Software  Streams 


Software-Software  Streams 


System  Increment 


>  Software  Increment 
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Software  Increment 


Hardware  Increment 

Module  B 

Module  A 
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IA  IA 
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IA 

9 


IA 

9 


IA 
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•  Specific  Challenges  Addressed 

-  Design  of  interfaces  and  the  tuning  of  Technical  Performance  Measures 
(TPMs)  related  to  dependent,  concurrently  developed  components 

-  For  concurrent  engineering  process  streams,  the  determination  of 

•  Optimal  number  of  interactions  between  concurrent  streams,  and 

•  The  optimal  place  of  interactions  in  the  life  cycle  (solved  by  using  APs) 


Synchronization  Via  Anchor  Points 


Pn4  Trailing  Process 


|  I 

Phase  . 

Phase  j+1 

P  n  Leading  Process 


1 

Phase  k 

Phase  k+1 

How  Anchor  Points  are  used 

-  Concurrent  process  streams  should  not  be  arbitrarily  shifted  or  overlapped 

-  Connection  is  only  planned  at  Anchor  Points 

Stakeholders  of  the  process  streams  collaborate  at  Anchor  Points 

-  Pn_f  stakeholders  rely  on  Pn  stakeholder  deliveries  at  APX  to  satisfy  APY 
objectives 


Example  Use  of  Anchor  Points  for  a  Product  Line 


4 

Product 

V 

>  <2 

->Product2 

1 

> 

M 

k  V 

XU 

...  l[»Tr» 

Platform 

What  is  a  product  line? 

-  A  product  line  represents  a  product  family,  a  set  of  related  systems  that 
are  built  from  and  leveraging  off  a  common  set  of  core  assets* 

Product  line  challenges 

-  Technical  considerations  -  selecting/distributing  product  features 

-  Business  constraints  -  balancing  cost  and  Time-to-Market 

-  Development  strategy  challenges  -  determination  of  architectural 
structuring,  development  and  production  order 

LCM  Challenge:  Manipulating  a  complex,  aggregate  life  cycle  model 


*  These  core  assets  are  also  called  the  elements  of  the  Product  Line  Platform 


DSM  (Design  Structure  Matrix) 

•  The  DSM  method  is  widely  used  to  design  and  optimize  complex 
systems  in  various  domains 

-  DSM  describes  the  relationships  between  architectural  elements  of  a 
system  in  a  concise  format 

-  In  each  cell  we  might  have  simply  a  marker  (like  a  circle)  or,  in  more 
complex  cases  some  kind  of  indicator  characterizing  the  relationship 
between  system  design  elements 

-  A  wide  range  of  tools  are  available  to  manipulate  DSM  [Browning  2001] 

•  Basic  DSM  Examples: 

D1  D2  D3  D4  D1  D2  D3  D4 


D1 

• 

D1 

D2 

• 

D2 

D3 

D3 

D4 

• 

D4 

Legend:  D1 ...  D4-  System  Design  Elements; 

m^  -  Relationship  between  D ,  and  Dj  Elements 
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Mapping  Anchor  Points  to  DSM 
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Concurrent  Increment  Coupling  Metric  (CICM) 


•  Coupling  is  a  measure  of  strength  of  interconnection 

-  Uncoupled  modules  are  independent 

•  High  or  Low  coupling  is  not  “good”  or  “bad” 

-  Various  pro’s  and  con’s  are  associated  with  different  coupling  levels 

-  Author’s  hypothesis  is  that  for  any  concurrent  engineering  situation 
an  optimal  coupling  exists 

•  DSM-based  CICM  definition 

CICM  =  m12 . mM, mnn) 

•  For  the  shown  DSM  matrices  a  simple  CICM  definition 

CICM  =  4  -  mnn 

where  mnn  is  the  value  from  the  diagonal  of  the  matrix 


CICM  Values  for  the  DSM  Examples 


mnn 

CICM 

Numeric  Value 

Ordinal  Rating 

0 

4 

Very  High 

1 

3 

High 

2 

2 

Medium 

3 

1 

Low 

0 

No  coupling  (Independent) 

The  Relationship  Between  CICM  and  Schedule/Cost  Risk 


•  Definitions 

-  Schedule  risk  in  this  context  is  risk  to  complete  the  project  in  the 
estimated  timeframe  due  to  unexpected  rework 

-  Cost  rjsk  in  this  context  is  risk  to  complete  the  project  within 
estimated  cost  due  to  unexpected  rework 

•  A  main  source  of  these  risks  is  architecture  volatility  stemming 
from  concurrent  engineering 

-  However ,  the  relationship  between  concurrent  increment  process 
stream  coupling  and  architecture  volatility  is  not  straightforward 

-  For  example,  the  classic  “Iron  Triangle ”  of  Cost-Schedule- 
Performance  does  not  apply  anymore 

•  Depending  on  the  chosen  concurrency  configuration  of  the 
increments,  drastically  different  schedules  are  expected  even 
though  performance  and  cost  are  supposed  to  stay  the  same 
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Discussion  Based  on  the  Examples 


•  Very  High  Coupling  (CICM=4) 

-  Positive: 

•  Increment  phases  overlap,  all  APs  are  aligned 

•  The  architecture  of  both  increments  is  basically  planned  together,  at  the 
same  time 

-  Being  able  to  change  both  architectures  provides  flexibility  that  is 
considered  positive 

•  This  configuration  promises  the  shortest  schedule 

-  Caveats: 

•  Both  architectures  are  volatile 

•  No  “hardening”  provided  for  the  leading  increment 

•  No  learning  from  the  development  of  the  leading  increment 

•  There  will  not  be  any  opportunity  for  early  detection  of  defects  in  the 
leading  increment 

-  This  configuration  results  in  the  most  costly  rework 


23 


Discussion  Based  on  the  Examples  (Cont.) 


•  High  Coupling  (CICM=3) 

-  Positive: 

•  Architectural  options  for  the  leading  increment  are  known 
when  the  design  of  the  trailing  increment  starts 

•  Actual  architecture  of  the  leading  increment  is  known  when 
the  determination  of  the  trailing  increment  architecture  starts 

•  The  actual  code  of  the  leading  increment  is  available  when 
the  implementation  of  the  trailing  increment  starts 

-  Caveat: 

•  Increased  cost  of  rework  when  correcting  any  problems  with 
the  leading  increment  that  are  discovered  during  the  design  of 
the  trailing  increment 


Discussion  Based  on  the  Examples  (Cont.) 


•  Medium  Coupling  (CICM=2) 

-  Positive: 

•  Actual  architecture  of  the  leading  increment  is  known  when 
the  work  on  the  trailing  increment  starts 

•  The  actual  code  of  the  leading  increment  is  available  when 
the  architectural  design  of  the  trailing  increment  starts 

-  Caveats: 

•  Increased  difficulty  in  correcting  any  problems  with  the  leading 
increment  that  are  discovered  during  the  design  of  the  trailing 
increment  due  to  the  fact  that  the  leading  increment’s 
architecture  has  been  determined 

•  Final  integration  is  further  removed;  correcting  any  problems 
with  the  leading  increment  that  are  discovered  during  final 
integration  is  becoming  increasingly  more  expensive 


Discussion  Based  on  the  Examples  (Cont.) 


•  Low  Coupling  (CICM=1) 

-  Positive: 

•  The  actual  code  of  the  leading  increment  is  available  when 
the  planning  of  the  trailing  increment  starts 

•  Leading  increment’s  code  is  considered  sufficiently  tested 

-  Caveats: 

•  High  level  of  difficulty  in  correcting  any  problems  with  the 
leading  increment  that  are  discovered  during  the  development 
of  the  trailing  increment  due  to  the  fact  that  the  leading 
increment  has  already  been  coded  and  tested 

•  Final  integration  is  further  removed;  Correcting  any  problems 
with  the  leading  increment  that  are  discovered  during  final 
integration  is  becoming  very  expensive 


Next  Steps  -  Direction  of  Future  Research 


•  Extend  CICM  to  cover  more  realistic  increment  positioning  situations 

-  The  shift  involves  more  than  one  phase 

-  Phase-lengths  are  not  equivalent 

•  Define  LCPC  (Life  Cycle  Plan  Cohesion)  Metric 

-  Cohesion  is  a  measure  of  how  tightly  bound  or  related  the  concurrent 
increments  are  to  one  another 

-  Coupling  is  one  key  factor ,  but  not  the  only  factor 

•  It  seems  to  be  plausible  that  tightly  coupled  increments  create  a  life 
cycle  plan  with  high  cohesion 

-  However,  the  relationship  needs  to  be  researched  and  quantified. 


Develop  quantitative  evaluation  guidance  for  LCPC 

-  Quantify  metrics 

-  Develop  a  methodology  that  allows  the  comprehensive  evaluation  of 
schedule,  rework,  and  quality  dimensions  of  different  life  cycle  plans 


Summary 


•  A  promising  Aerospace  research  platform,  ULCM®  has  been 
used  to  model  concurrent  engineering  process  streams  of 
software-intensive  system  development 

•  DSM  has  been  introduced  to  facilitate  the  easy  manipulation 
of  ULCM®  models  of  concurrently  engineered  complex 
systems 

•  Two  new  metrics,  CICM  and  LCPC  has  been  proposed  to 
characterize  increment  coupling  and  cohesion  in  complex 
life  cycle  models 


Acronyms 


AP 

Anchor  Point 

CICM 

Concurrent  Increment  Coupling  Metric 

DSM 

Design  Structure  Matrix 

IDR 

Increment  Delivery  Readiness 

IED 

Increment  End-of-Life  Decision 

IEL 

Increment  End-of-Life 

MR 

Increment  Inception  Readiness 

ILA 

Increment  Life  Cycle  Architecture 

ILO 

Increment  Life  Cycle  Objectives 

IOR 

Increment  Operational  Readiness 

IR&D 

Independent  Research  &  Development 

LCM 

Life  Cycle  Modeling 

LCPC 

Life  Cycle  Plan  Cohesion 

MOIE 

Mission-Oriented  Investigation  and  Experimentation 

TPM 

Technical  Performance  Measure 

ULCM 

Unified  Life  Cycle  Modeling 
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-  Why 

-  How 
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-  Excerpts 


Today  is  a  Discussion  not  a  Lecture  -  Please  Stop  me  Anytime! 
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Direction  &  Goals 


•  In  2006,  Tasked  to: 

-  Perform  a  Center-wide 
SE  Assessment 

-  Found  Out 
Where  We  Are? 

-  Baseline  Enterprise 
Process  Improvement 


•  Goals 

-  Improve  Program 
Performance  &  Reduce 
Technical  Risk 

-  Ensure  a  Consistent 
Understanding  of  SE 

-  Ensure  Core  SE 
Processes  are  in  Place 
and  Being  Practiced 

•  Identify  Opportunities  for 
Continuous  Improvement 

•  Clarify  Roles  and 
Responsibilities 

-  Institutionalize  “Best 
Practices” 
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Why  We  Need  Change? 


Too  Many  Problems  Have  Surfaced 

-  Missed  or  Poorly  Validated  Requirements 

-  Poor  Planning  Fundamentals 

-  Lack  of  Integrated  Risk  Management 

-  Lack  of  Rigorous  Process 

-  Lack  of  Process  Flow  Down 

We  Must  Regain  Our  Credibility 

Restoring  SE  Discipline  in  AAC 
Projects  Is  a  Key  Initiative 


Lack  of  Disciplined  Systems  Engineering 
has  been  a  Major  Contributor  to  Poor  Program  Performance 
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Our  Approach 


ifiMf 


*  Define  Systems  Engineering  Best  Practices 


*  Benchmark  Systems  Engineering  Implementation 


*  Establish  a  Baseline  for  Continuous  Improvement 
-  Begin  Changing  the  Culture  to  Kaizen 


•  Phased  Approach  -  3  Phases 
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SE  Models  &  Frameworks 


PSP 

People 
CMM 

SA-CMM 


SDCCR 


MIL-Q  MIL-STD-1679 

STD- 
2168 


EIA/IEEE 

J-STD-016 


SECAM 

/ 

IEEE 

1220  EIA/IS 

MIL-STD  _ _  632 

-499 B 


SSE- 

CMM 


ISO  15288 


IEEE/EIA 

12207 
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Defining  SE 


•  Center 
Engineering 
Steering 
Council 

-  Defined 
Criteria 

-  Approved 
Module  & 
Approach 


CM  Ml®  Acquisition 
Module  (CMMI-AM), 
Version  1.0 


Tom  Bernard,  USAF  ASC/EN 
Brian  Gallagher,  SE! 

Roger  Bate,  SEi 

Hal  Wilson,  Northrop  Grumman 


February  2004 


Best  Practices 

r  

- - i 

MIL-STD-499B  \  - 

J  1_ 1_ 1 

EIA  632 

Ijj  AAC 

-I  1  1 

INCOSE  j 

ISO  15288  | 

•  9  Key  Process 

Areas 


CMMI-AM 
.  ail-arid  ice 
Air  Ancamni  CHm 
Syvrafi  Eh  213  “tins 

VEJiLon  l  .C 
*5  Vlmr  XMfi 


May  2006 


29  Goals 
117  Practices 

9  Generic 
Practices 

Qualifying 

Questions 

43  Pages 


Industry/ Academia 


•  SEI,  NDIA,  Boeing, 
Raytheon,  etc. 

•  USC,  AFIT,  etc. 


OSD  Guidance 

AF  Guidance 

AFMC  Guidance 

Other  Centers 

•  DAG 

•  AFI  63-1201 

•  AFMCI  63-1201 

•  ESC 

•  SEP  Guidance 

•  AFPD  OSS&E 

•  OSS&E 

•  SMC 

AAC  Assessment  Module  Based  on  International,  Industry  and  DoD  Best  Practices 
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Benchmarking  the  Enterprise 


Process  Area  Criteria* 
O  >90%  of  Practices 
O  65-89%  of  Practices 
•  <65%  of  Practices 


Program  Criteria 
O  >90%  of  Practices,  No  Red 
O  65-89%  of  Practices,  NTE  1  Red 
•  <65%  of  Practices,  2  or  More  Ret 


:  Weighting 

SPs75% 
GPs  25% 


Key  Process  Areas 


as  of  8  Jan  07 

R 

D 

T 

P 

TA 

RM 

CM 

1  Pgm  1 

Program  #1 

Program  #2 

Program  #3 

Program  #4 

1 

Program  #5 

1 

Program  #6 

Program  #7 

Program  #8 

Program  #9 

Program  #10 

Program  #1 1 

Program  #12 

Program  #13 

Program  #14 

Program  #15 

Program  #16 

Program  #17 

Program  #18 

1  1 

1  1 

Center  Average 

Portfolio  Criteria 
O  95%  Programs  Green 
O  75%-95%  Programs  Green,  <10%  Programs  Red 
•  <75%  Programs  Green  or  >10%  Programs  Red 
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V  AAC  SE  Assessment  Model 


*  Engineering  Council 
Provided  Steering 

*  Working  Level  Team 
Did  Heavy  Lifting 

-  Defined  SE 

-  Established 
Expectations 

-  Facilitated 
Assessments 

-  Training  Benefits 


Systems  Eitgmcermg 
A aseasmeiif  K  lode  I 
i  SEAM*  Version  1  0 

Oct  2007 


AAC  Practices 

- — 


MIL-STD-499B 


j  i  ii 


EIA  632 

»  i  i 


INCOSE 


L 


CMMI 


ISO  15288  | 


Industry/ Academia 

OSD  Guidance 

AF  Guidance 

•  SEI,  NDIA,  Boeing, 

Raytheon,  etc. 

•  DAG 

•  AFI  63-1201 

•  USC,  AFIT,  etc. 

•  SEP  Guidance 

•  AFPD  OSS&E 

/ - / 

AAC-SEAM  V2.09 

•  10  Process  Areas 

•  34  Specific  Goals 

•  120  Practices 

•  6  Generic  Practices 

•  Qualifying  Questions 

•  50  Pages 


AFMC  Guidance 

Other  Centers 

•  AFMCI  63-1201 

•  ESC 

•  OSS&E 

•  SMC 

_ — — 

AAC  Assessment  Model  Based  on  International,  Industry  and  DoD  Best  Practices 
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Process  Area  Evolution 


Gmmm 


I 


•  Technical  Processes 

-  Requirements 

-  Design 

-  Verification/Validation 

-  Transition 


•  Technical  Management 
Processes 

-  Planning 

-  Risk  Management 

-  Configuration  Management 

-  Decision  Analysis 

-  Technical  Assessment 


•  Technical  Processes 

-  Requirements 

-  Design 

-  Manufacturing 

-  Verification/Validation 

-  Sustainment 

•  Technical  Management 
Processes 

-  Planning 

-  Risk  Management 

-  Configuration  Management 

-  Decision  Analysis 

-  Technical  Assessment 


Consistent  with  OSD  Policy,  Defense  Acquisition  Guidebook, 
Draft  AFI  on  Systems  Engineering  &  AFMCI 63-1201 
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Collaboration  &  Refinement 


A 


2007-2008  Goals 

-  Reduce  Burden  on 
Execution 

-  Refine  Alignment  Between 
Module  and  DoD,  AF, 
AFMC  Guidance 


AAC-SEAM  v2.4 

•  10  Process  Areas 

•  33  Specific  Goals 

•  115  Practices 

•  7  Generic  Practices 

•  67  Qualifying  Questions 

•  47  Pages 


•  Formed  AAC  Tiger  Team  to 
Work  on: 

-  Streamlining 

-  Expanded  Coverage 

•  Collaboration  with  OSD  and 
Software  Engineering 
Institute  on  Future  of  CMMI 

•  AF  Wide  Collaboration  to 
Develop  Common  SEA  Model 

•  Industry  Collaboration 


Compliant  with  AF-SEAM  vl.  0 
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Current  Process  Areas 


Technical  Process  Areas 

-  Requirements 

-  Design 

-  Manufacturing 

-  Verification  &  Validation 

-  Fielding  &  Sustainment 

Project  Process  Areas 

-  Project  Planning 

-  Risk  Management 

-  Configuration  Management 

-  Decision  Analysis 

-  Technical  Assessment 


-  Introduction 

-  Goal 

•  Practices 

•  Grey  Matter 

*  Question(s) 

-  Goal... 

*  Generic  Practices 

-  Question(s) 


AAC-SEAM  v2.4 
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Requirements  Process  Area 


•  Purpose:  Develop  and  analyze  operational  user,  product,  and 
product-component  requirements 

•  Goals: 

-  RG1:  Stakeholder  needs,  expectations,  constraints,  and  interface 
requirements  are  collected  and  translated  into  a  definition  of  needed 
product  capabilities/characteristics 

-  RG2:  Requirements  are  refined,  elaborated  and  allocated  to  support 
product  design 

-  RG3:  Iteratively  analyze  and  validate  operational  and  derived 
requirements  throughout  the  product  life  cycle 

-  RG4:  Requirements  are  managed  and  controlled,  and  inconsistencies 
with  technical  plans  and  work  products  are  identified 

-  RG5:  Generic  practices  are  applied  to  the  requirements  process  area 

•  13  specific  &  7  generic  practices  to  be  assessed 
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Example  Practice 


Key  Process  Area:  Requirements 

Goal:  RG4  -  Requirements  are  managed  and  controlled,  and 
inconsistencies  with  technical  plans  and  work  products  are 
identified. 

Practice: 

PI  Use  a  disciplined  process  for  accepting,  vetting,  approving  and  providing 
requirements  and  changes  to  the  developer  through  a  single  focal  point. 


This  process  should  prevent  developers  from  receiving  requirements  changes 
from  unauthorized  sources  that  are  outside  the  flow  of  the  acquirer’s 
established  configuration  management  process.  Each  change  to  a  controlled 
requirement  should  be  assessed  for  impact  to  the  program’s  performance, 
cost,  and  schedule  baselines  and  to  program  risk.  The  existing  cost,  schedule, 
and  performance  baselines  should  be  changed,  as  required,  to  accommodate 
the  requirements  change.  “Requirements  creep”  must  be  avoided.  A  new 
requirement  must  be  backed  with  money  and  vetted  through  a  control  process. 


Self  Assessment  Consists  of  Answering  Yes,  No  or  Not  Applicable 
with  Supporting  Rationale  to  each  Practice  -  No  Partial  Credit 
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Requirements 

- -APMM 

•  Design  Mission  Reference  Profiles  (RG1P2) 

-  Comprehensive  Definition  of  Product  Characteristics  in 
Engineering  Terms  and  Documentation  of  the  Interaction  of 
the  Product  with  the  Environment,  Other  Systems,  and 
Operational  Users  [Willoughby]. 

Do  we  understand  the  edges  of  the  technical  performance  envelope? 

•  Validate  Requirements  (RG2P3) 

-  Ensure  the  Evolving  Product  will  Perform  as  Intended  in  the 
Operational  Environment  [CMMI]. 

Do  the  derived  requirements  accurately  and  completely  represent  what 
is  needed?  and  no  more. . .  How  were  they  validated? 


Reference:  AAC  SEAM  v2.4 
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Planning 


*  Integrated  Plans  for  Managing  (PPG2) 

-  Plan  for  Engineering,  Data,  Resources,  Stakeholders, 
Technology,  Reliability  and  Supportability.  Maintain 
Integrated  Master  Schedule  and  Plans  [CMMI]. 

Are  all  technical  plans  integrated  and  consistent?  How  do  you  know? 

•  At  the  fundamental  level,  planning  includes 

understanding  what  must  be  done  (scope  of  effort), 
who  needs  to  do  it  (staffing  and  skills),  when  it 
needs  to  be  done  (life  cycle  and  schedule),  how  it 
is  to  be  done  (reviews,  methodology,  tools, 
meetings  etc...)  and  how  much  it  will  cost. 


Reference:  AAC  SEAM  v2.4 
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Manufacturing 

- -APMM 

•  Plan  for  Transition  to  Production  (RG1P2) 

-  Establish  Comprehensive  Management  Plans  that  Describe 
All  Production  Related  Activities  that  Must  Be  Accomplished 
During  Design,  Test  and  Low-rate  Initial  Production 
[Willoughby]. 

Are  all  tiers  of  suppliers  are  involved  in  production  planning? 

•  Implement  Quality  Management  (MG4P1) 

-  Monitor  and  Control  Manufacturing  Processes  and  Product 
Variation  in  all  Tiers  of  Manufacturing  [Willoughby]. 

How  are  process  changes  considered ,  authorized  and  implemented? 


Reference:  AAC  SEAM  v2.4 
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Testing  and  Sustainment 


immm 


•  Verification  and  Validation  (VG3  &  VG5) 

-  Analyze  and  document  the  results  of  the  verification  & 
validation  activities,  identify  issues,  initiate  and  document 
corrective  actions  [CMMI]. 

Is  information  on  issues  and  corrective  actions  widely  known? 


•  Plan  for  Logistic  Support  (SGI PI) 

-  Comprehensive  Life  Cycle  Plan  for  Ensuring  a  Safe,  Suitable 
and  Effective  Product  [AFMCI]. 


Are  the  critical  failure  modes  addressed  in  maintenance  activities? 


Reference:  AAC  SEAM  v2.4 
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Decision  Analysis 


■AFMm 


*  Guidelines  For  Decision  Making  (DAG1P1) 

-  Determine  Which  Issues  Are  Subject  To  Formal  Evaluation 
[CMMI]. 


Do  we  understand  when  a  formal  analysis  of  alternative  courses 
is  indicated?  Do  we  have  the  discipline  to  comply?  . . . 

•  Document  Decision  Rationale  (DAG1P6) 

-  Including  Dissenting  Opinions  [NASA] 

-  Support  Future  Analysis  [CMMI] 

Have  we  documented  the  decision  including  any  concerns/issues?  ... 
Sufficiently  to  support  a  re-examination  in  the  next  phase? 


Reference:  AAC  SEAM  v2.4 
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Final  Thoughts 

- jmmm 

“The  Air  Force  is  not  funded  to  do  everything  that  everybody  wants  us  to  do.  ” 

—  Hon  Michael  Wynne,  SECAF 

So  let’s  agree  in  AFMC  we  are  done  with  the  phrase  ffmore  with  less .  ” 

Instead,  I’d  like  us  each  to  focus  on  doing  the  right  things  with  the  available 
resources.  I  want  you  to  ask  yourself  the  question,  " with  the  resources  I  have  at 
my  disposal  (time,  funding,  people,  equipment,  etc.)  what  are  the  most 
important  things  I  have  to  do?  "  The  corollary  question  then  becomes,  "what 
must  I  stop  doing  so  I  can  do  those  things?  "  I  recognize  there  are  valuable 
things  you  might  have  to  stop  doing.  I  need  each  of  you  to  take  a  hard  look  at 
your  organization  and  determine  what  those  things  are.  ” 

—  Gen  Bruce  Carlson,  Commander  AFMC 


This  is  Great  News  for  Systems  Engineering  Because  we  are  All  About  Optimizing  Systems ! 
but  We  Must  Have  the  Discipline  and  the  Integrity  to  Make  the  Trades... 
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Kai-zen 

The  Art  of  Continuous  Improvement 

Kai-zen  must  operate  with  three  principles  in  place: 
process  and  results,  systemic  thinking,  and  non-blaming 

(because  blaming  is  wasteful). 


Air  Armament  Center 


War-Winning  Capabilities... On  Time,  On  Cost 


Lessons  Learned  Doing 
Systems  Engineering 
Assessments  on  the 
Government 

Ian  Talbot 

AAC/EN 

ian.talbot@eglin.af.mil 

https://afkm.wpafb.af.mil/EglinSE 


DISTRIBUTION  STATEMENT  A:  Approved  for  public  release;  distribution  is  unlimited.  96  ABW/PA  #10-17-08-473 


Product  Portfolio 
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Outline 


-AFMM 


•  Air  Armament  Center 
Systems  Engineering 
Assessments 

-  Why 

-  How 

-  What  we  Learned 

-  Futures 


Today  is  a  Discussion  not  a  Lecture  -  Please  Stop  me  Anytime! 
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Direction  &  Goals 


•  In  2006,  EN  Tasked  to: 

-  Perform  a  Center-wide 
SE  Assessment 

-  Found  Out 
Where  We  Are? 

-  Baseline  Enterprise 
Process  Improvement 


•  Goals 

-  Improve  Program 
Performance  &  Reduce 
Technical  Risk 

-  Ensure  a  Consistent 
Understanding  of  SE 

-  Ensure  Core  SE 
Processes  are  in  Place 
and  Being  Practiced 

•  Identify  Opportunities  for 
Continuous  Improvement 

•  Clarify  Roles  and 
Responsibilities 

-  Institutionalize  “Best 
Practices” 
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Our  Approach 


*  Define  Systems  Engineering  Best  Practices 


*  Benchmark  Systems  Engineering  Implementation 


*  Establish  a  Baseline  for  Continuous  Improvement 
-  Begin  Changing  the  Culture  to  Kaizen 


•  Phased  Approach  -  3  Phases 
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Quality 


Focus  on  Process 


•  The  Quality  of  a  System  or  Product  is 
Highly  Influenced  by  the  Quality  of  the 
Process  Used  to  Develop  and  Maintain  It 


Ad  Hoc  Repeatable  Defined  Managed  Optimized 


Process  Discipline  - ► 


CMMI  Performance  Resul 

ts  Summary 

Median 

Improvement 

Number  of 
Data  Points 

Cost 

34% 

29 

Schedule 

50% 

22 

Productivity 

61% 

20 

Quality 

48% 

34 

Customer 

Satisfaction 

14% 

7 

ROI 

4.0  :  1 

22 

CMU/SEI-2006-TR-004 

Process  Discipline 
Leads  to: 

-  Predictable  Program 
Performance 

-  Ability  to  Deliver  on 
our  Commitments 


Institutionalized  Process  Driven  SE  »  Lower  Risk  Technical  Programs 
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AAC  SEA  Model  Development 


1 


CMMI®  Acquisition 
Module  (CMMI-AM), 
Version  1,0 


B 


CKMI-AM 
.  QL-artd  iar 
A\z  Anndoan  C.e 

SyVFSIE 

Ic^LememnoE 

A;=eE.ais2n 


30  + 

Assessments 


May 
2006  i] 


Systems  Engineering 
Assessment  Model 
(SEAM)  Vernon  2  0 

I  October  2tK)7 


Oct 

2007 


Ir-ahrWlffl  C-CTr-jil 
E>nl*«d.  ■Vt/W  \J.\  m\ 


Lut.  fy  ■*  i*nJ? 

r.3^. 


ikt 

JTWWUfi . .  n 

ir»  Ht  Mitsui 


hrl-flUtt*  rinvti/TLM 
Lb<J  .'BdliKlfl 


Pa I  "--tt. . 


Industry/ Academia 
SEI,  NDIA,  Boeing, 
Raytheon,  etc. 

USC,  AFIT,  etc. 


* 


Systems  Engineering 
Assessment  Model 

Aug  2008 


CVifit^w  L"jj-  ,v  —l 1  *rUW 


■■  fn 


AAC  Practices 

- — 


MIL-STD-499B 


j  i  ii 


EIA  632 

»  i  i 


INCOSE 


L 


CMMI 


Systems  Engineering 
Assessment  Model  v2.4 

•  10  Process  Areas 

•  33  Specific  Goals 

•  115  Practices 

•  7  Generic  Practices 

•  67  Qualifying  Questions 

•  47  Pages 


ISO  15288  | 


Streamlined 
CMMI 


Compliant  with 
AF-SEAM  vl.O 


OSD  Guidance 

AF  Guidance 

AFMC  Guidance 

Other  Centers 

•  DAG 

•  AFI  63-1201 

•  AFMCI  63-1201 

•  ESC 

•  SEP  Guidance 

•  AFPD  OSS&E 

•  OSS&E 

•  SMC 

AAC  Assessment  Model  Based  on  International,  Industry  and  DoD  Best  Practices 
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Current  Process  Areas 


Technical  Process  Areas 

-  Requirements 

-  Design 

-  Manufacturing 

-  Verification  &  Validation 

-  Fielding  &  Sustainment 

Project  Process  Areas 

-  Project  Planning 

-  Risk  Management 

-  Configuration  Management 

-  Decision  Analysis 

-  Technical  Assessment 


-  Introduction 

-  Goal 

•  Practices 

•  Grey  Matter 

*  Question(s) 

-  Goal... 

*  Generic  Practices 

-  Question(s) 


AAC-SEAM  v2.4 
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Criteria  for  Methodology 


•  Objective  Assessment 

•  Provide  insight  into 
Government,  Prime 
Contractors  and  Subs 
Process  &  Capability 

*  Facilitate  Self  Assessment  & 
Continuous  Improvement 

-  Lean  &  Six  Sigma 

*  Consistent  Near  and  Far  Term 
Approach 


•  Provide  Results  that  are 
meaningful  for  leadership 

-  Relevant  to  PM/PEO 

-  Simple 

-  Understandable 

-  Graphical 

*  Support  Multi-level 
Measurement  &  Reporting 

-  Program,  Group,  Wing, 
Enterprise 
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SEA  Methodology 


Acquirer  &  Supplier 


SEA  Team 
Peer  Review 


Project  Team 

Leadership 

Self-Assessment 

W 

Review  Board 

Team  Chaired  by  Senior  Systems  Engineer 
Members  from  Across  Multiple  Programs 


Co-chaired  by  Chief  of  Systems 
Engineering  and  Line  Engineering 
Functional 


Assessment  Process  Time  Required 
Leadership  -  8  person  hrs 
Project  Team  -60-100  person  hrs 
SEA  Team  -  <50  person  hrs 


SEA  Assess  What  Practices  are  Implemented  NOT  How  Well  Executed 
Future:  Begin  to  Shift  Focus  to  “How  To  ”  and  Quality  of  SE  Implementation 


080806  SEA  Lessons  Learned;  Talbot 


Products  Provided  to  Program 


•  Training  &  Self 
Assessment 

•  Peer  Review 
Collaboration  & 
Feedback 


•  Validated  Assessment 

•  Summary  Memorandum 

-  Findings  &  SE  Improvement 
Recommendations 
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Benchmarking  the  Enterprise 


Process  Area  Criteria* 
O  >90%  of  Practices 
O  65-89%  of  Practices 
•  <65%  of  Practices 


Program  Criteria 
O  >90%  of  Practices,  No  Red 
O  65-89%  of  Practices,  NTE  1  Red 
•  <65%  of  Practices,  2  or  More  Ret 


:  Weighting 

SPs75% 
GPs  25% 


Key  Process  Areas 


as  of  8  Jan  07 

R 

D 

T 

P 

TA 

RM 

CM 

1  Pgm  1 

Program  #1 

Program  #2 

Program  #3 

Program  #4 

1 

Program  #5 

1 

Program  #6 

Program  #7 

Program  #8 

Program  #9 

Program  #10 

Program  #1 1 

Program  #12 

Program  #13 

Program  #14 

Program  #15 

Program  #16 

Program  #17 

Program  #18 

1  1 

1  1 

Center  Average 

Portfolio  Criteria 
O  95%  Programs  Green 
O  75%-95%  Programs  Green,  <10%  Programs  Red 
•  <75%  Programs  Green  or  >10%  Programs  Red 
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Lessons  Learned 


•  Personnel  Resources  are  Stretched  and  Need 
SE  Training  &  Experience 

•  Process  and  Procedures  are  Needed  to 
Ensure  More  Repeatable/Consistent 
Application  of  SE 

•  Product  Line  Specific  Guidebook 
Capturing  Eglin  Experience  in  Weapons 
Desired 
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The  Good 


*  Requirements  Control  & 
Verification  Working 
Group 

*  Iterative  Requirements  & 
Design  Trade-off 
Working  Group 

*  Concurrent  Engineering 
to  Ensure  Successful 
Transition  to  Production 


*  Contract  Incentives  for 
Reducing  Cost  and 
Increasing  Reliability 

*  Full  Trust 
Integrated  Teaming 

*  Integrated  & 
Overarching  Risk 
Management  Strategy 


“Following  MIL-STDs  was  Better  than  Having  No  Process  at  All” 
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The  Bad 


Legend 


R  -  Requirements  V  -  Ver/Val  P  -  Planning  CM  -  Config  Mgmt  TA  -  Tech  Assessment 

D  -  Design  T  -  Transition  RM  -  Risk  Mgmt  DA  -  Decision  Analysis 


real 


Weakness 

CD 

Strength 


/  R  \| 

D 

V 

T 

1/  P  \ 

TA 

RM 

CM 

_ 1 

/ 

/ 

1 

1 

\ 

\ 

\y 


Areas  that  Need  Work 

-  Requirements 

-  Decision  Analysis 

-  Planning 

-  Process  Integration  Particularly 
Risk  Management 

Model  Expansion  Needed 

-  Manufacturing  (Transition  to  Production) 

-  Sustainment 

Added  in  Version  2.0 


Decision  Analysis 

RED 

Planning 

YELLOW 

Requirements 

Risk  Management 

Verification  &  Validation 

Transition 

Technical  Assessment 

Design 

Configuration  Management 

GREEN 

080806  SEA  Lessons  Learned;  Talbot 


15 


Better 


Requirements  Weaknesses 


•  Design  Mission  Reference  Profiles  (RG1P2) 

-  Comprehensive  Definition  of  Product  Characteristics  in 
Engineering  Terms  and  Documentation  of  the  Interaction  of 
the  Product  with  the  Environment,  Other  Systems,  and 
Operational  Users  [Willoughby]. 

Do  we  understand  the  edges  of  the  technical  performance  envelope? 

•  Validate  Requirements  (RG2P3) 

-  Ensure  the  Evolving  Product  will  Perform  as  Intended  in  the 
Operational  Environment  [CMMI]. 

Do  the  derived  requirements  accurately  and  completely  represent  what 
is  needed?  and  no  more. . .  How  were  they  validated? 


Reference:  AAC  SEAM  v2.4 
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Some  Solutions 


iWM 


•  Develop  Valid  Mission  Reference 
Profiles  to  Support  Design 

-  Validate  Concepts  of  Employment 


-  Obtain  Accredited  Simulation  Capability 
Including  Carriage,  Separation,  Fly-out 

•  Engage  Independent  Subject  Matter  Experts 

•  Discover  &  Examine  Stressing  Conditions 


Vibration 

Acoustics 


-  Anchor  the  Models  with  Data 

•  Test  Prototypes  in  Wind  Tunnel 

•  Test  Instrumented  Flight  Vehicles  in 
Carriage,  Separation  and  Fly-out  Modes 


Temperature 

Electromagnetic 

Aerodynamic 


•  Test  Sample  Conditions  of  All  Configurations 
With  Representative  Hardware  Early  and 
Allow  Schedule  for  Issue  Resolution 
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Sustainment  Weaknesses 


•  Establish  Operational,  Suitability  and 
Effectiveness  Baselines  (SG4P1) 

-  Conduct  Health  Monitoring  and  Verification  to  Ensure  Fielded 
Product  Matches  Baseline  Performance  [AFMCI] 

How  do  we  assure  the  products  continued  safety  &  performance? 


•  Perform  Audits  to  Maintain  Integrity  (CMG3P2) 

-  Ensure  Processes  for  Maintaining  the  Integrity  of  the  Fielded 
Configuration  are  Effective  [CMMI]. 

How  do  you  know  if  Time  Critical  Technical  Orders  are  compete? 


Reference:  AAC  SEAM  v2.4 
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AF-SEAM  Background 


*  In  2006,  USAF  Material  Command  Engineering 
Council  Action  Item  to: 

-  Provide  an  USAF-wide  SE  Assessment  Model 

-  Involve  USAF  Centers  (product  and  logistics) 

-  Leverage  current  CMMI®-based  models  in  use  at  AF 
Centers 

-  Baseline  Process  Capability  &  Usage 

*  AF  Systems  Engineering  Assessment  Model: 

-  A  single  AF-wide  tool  which  can  be  used  for  the 
assessment  and  improvement  of  systems  engineering 
processes  in  a  program/project. 
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AF-SEAM  SP  Roll-Up 


Specific  Practice  Assessment  Results 

XXX  Center 


CM  DA  D  M  PP  R  RM  S  TMC  V 

Process  Area 
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AF-SEAM  GP  Roll-Up 


Generic  Practice  Assessment  Results 

XXX  Center 


GP1  GP2  GP3  GP4  GP5  GP6  GP7 

Practice  Area 
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Future  Concept 


Key  Process  Area:  Manufacturing  or  TMC 

Goal:  -  Product  and  process  quality 
is  assessed  and  improved. 

Practice: 

PI  Establish  and  maintain  a  quality  management  system. 


5:  The  developer  and  major  suppliers  have  an  ISO  9000/AS9100 
certified  operation  with  recent  AS9 101  audit  at  relevant  locations. 

4:  The  developer  has  an  ISO  9000/AS9100  certified  operation 
with  recent  AS9101  audit  at  relevant  locations. 

3:  The  developer  is  meeting  the  intent  of  ISO  9000/AS9100  with  a 
recent  independent  quality  audit  at  relevant  locations. 

2:  The  developer  has  an  effective  quality  management  system  that 
includes  suppliers  with  no  recent  independent  audit. 

1:  The  developer  has  not  demonstrated  an  effective  quality 
management  system. 


Rungs  Facilitate  1)  Self  Assessment,  2)  Training  and  3)  Steps  for  Improvement 
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Summary 


•  Goal  is  to  Continue  to  Improve  Program  Performance 

-  Too  Many  Examples  of  Program  Performance/  Issues  Being 
Tracked  Back  to  Lack  of  Systems  Engineering  Discipline 

•  Long  Term  Goal  -  Revitalizing  Systems  Engineering 

-  Need  to  Follow  “Best  Practices” 

-  Need  to  Do  them  “Well” 

-  Need  to  Ensure  that  Our  Program  Teams  Have  What  they  Need 

•  Qualified  People,  Process  Discipline,  Tools/Technology 
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Kai-zen 

The  Art  of  Continuous  Improvement 

Kai-zen  must  operate  with  three  principles  in  place: 
process  and  results,  systemic  thinking,  and  non-blaming 

(because  blaming  is  wasteful). 


October  21 


ASN(RDA)  Chief  Systems  Engineer 
carl.siel@navy.mil 


Navy  Software  Process 
Improvement  Initiative  (SPII) 


IRda 

Chief 

Systems 

Engineer 


CMM  Level  im  Test'n9 

WCaWc*™"""'* 

ASN  (RDA)  Software  Policies 


ENVIRONMENT 


sfant/j 


o„  C#/,Jfe 

°Pment 

ranee  To  Change  °0rS 

Immature  Costing  Models 

External  Drivers 

°Cesses  For  d  Secu 

M«ndtr<lP«'“sses 

!ndll^6l%aSebi,ity  Concerns 


SSG:  Senior  Steering  Group 
HIT:  Horizontal  Integration  Team 
SAM:  Software  Acquisition  Management 
SSE:  Software  Systems  Engineering 
SWDT:  Software  Development  Techniques 
Bl:  Business  Implications 
HR:  Human  Resources 


Non- 

Enterprise  Licenses 


♦  Increase  leadership 
awareness  and 
accountability 

♦  Better  align  Naval 
acquisition  with  our 
industry  partners 

♦  Develop  a  skilled 
acquisition  force 

♦  Holistic  Systems 
Engineering  Approach 
focused  on  key  functional 
areas: 

-  Software  Acquisition 
Management 

-  Software  Engineering 
Practices 

-  Business  Implications 

-  Software  Development 
Techniques 

-  Human  Resources 


SPII  Charter:  15  May  2006  ASN  RDA  Memo 


The  Plan 


I.  As  Is: 

Understand 
current  situation  and 
review  existing 
policies  and  reports 


II.  To  Be: 

Envision  things 


to  come  & 
document  changes 


III.  Institutionalize: 
Leverage  existing 
Mechanisms; 
PEO  and  SYSCOM 
responsibilities 


[Rda 

Chief 

Systems 

Engineer 


Institutionalize 

Overarching  Policy  and  Guidebook  for  Acquisition  of  SW  Intensive  Systems 


Step-wise  Accomplishments 


IRda 

Chief 

Systems 

Engineer 


♦  As  Is”  Report  signed  17  May  2007 

-  Uncovers  the  current  environment  for  the  acquisition  of  software 
intensive  systems  across  the  Naval  Enterprise 

-  Findings  are  consistent  with  past  DSB  and  NRAC  findings 

♦  “Software  Development  Techniques  Phase  1  Report”  signed 
10  Jul 2007 

-  Provides  an  overview  of  existing  software  development  techniques  and 
suggestions  for  evaluating  emerging  software  development  techniques 

♦  Program  Office  Survey  Findings  Report  promulgated  July  2007 

-  Report  verifies  the  findings  of  previous  studies  (e.g.,  Defense  Science 
Board  (DSB)-2000  and  Naval  Research  Advisory  Committee  (NRAC)- 
2006)  by  tracking  them  directly  to  current  programs  of  record 

♦  Contract  Language  Guidance  policy  memo  signed 
13  Jul 2007 

-  Provides  amplifying  guidance  information  on  the  17  Nov  2006  Contract 
Language  policy  memo 
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Accomplishments  (cont.) 


IRda 

Chief 

Systems 

Engineer 


♦  Software  Metrics  White  Paper  -  identified  4  core  metrics 

♦  “To  Be”  Report  signed  6  Nov  2007 

-  Assists  acquisition  professionals  with  a  preview  of  key 
considerations  for  major  problems  having  been  found  to  be 
most  troublesome  and  most  commonly  documented 

♦  “Role  Base  Right  Fit  Training”  Report  signed 

6  Nov  2007 

-  Addresses  the  training  issues  highlighted  by  the  SAM  focus 
team  “As  Is”  state  report,  SSE  focus  team  “Program 
Management  Office  Survey  Findings,”  DSB,  and  NRAC 
findings 

♦  Contract  Language  policy  memo  signed  17  Nov  2006 

-  Directs  standardized  contract  language  for  all  contracts 
containing  software  development,  acquisition  and  life  cycle 
support  beginning  with  RFrs  issued  after  1  Jan  2007 

•  Requires  developers  to  submit  Software  Development  Plan  (SDP) 


5 


Core  Software  Metrics 


IRda 

Chief 

Systems 

Engineer 


♦  The  four  required  core  metrics 

-  Software  Size/Stability 

-  Software  Cost/Schedule 

-  Software  Quality 

-  Software  Organization 

♦  All  metrics  to  be  provided  during  key  phases  of  the 
system  acquisition  lifecycle  and  DoN  2Passes/6Gates 


ID 

Phase 

Milestone-Related  Period 

l 

Concept  Development 

Pre-Concept  Decision  (CD) 

ll 

Concept  Refinement 

Post-CD,  Leading  to  Milestone  (MS)-A 

ill 

Technology  Development 

Post  MS-A,  Leading  to  MS-B 

IV 

System  Development  and  Demonstration  (SDD) 
(System  Integration) 

Post  MS-B,  Leading  to  Design  Readiness  Review  (DRR) 

V 

SDD  (System  Demonstration) 

Post  DRR,  Leading  to  MS-C 

VI 

Production  and  Deployment 

Post  MS-C,  Leading  to  Full  Rate  Production  (FRP)  Decision 

VII 

Operations  and  Support 

Post  FRP  Decision  Review 
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Status  Reporting  Based  on  Metrics 


IRda 

Chief 

Systems 

Engineer 


♦  Examples  of  basic  and  general  usage  of  metrics: 

-Scope  creep  and  software  stability  based  on  software  size 
metrics/trends 

-Software  cost  and  schedule  variances,  trends,  and 
performance  indexes 

-Software  defects,  trouble  reports,  and  other  quality  trends 

-Software  personnel  staffing  actuals  vs.  planned,  including 
training  and  turnover  metrics 

♦  Software  4  Core  Metrics  infused  into  Naval 

Probability  of  Program  Success  (PoPS)  -  Complete 
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SPII  is  Institutionalized! 


IRda 

Chief 

Systems 

Engineer 


♦  Software  Process  Improvement  Initiative  completed  - 
Sept  2008 

-  Software  Measurement  for  Naval  Software  Intensive 
Systems 

•  4  core  metrics 

-  Overarching  Software  Process  Improvement  Policy  for 
Acquisition  of  Naval  Software  Intensive  Systems 

•  Software  Process  Management  Improvement 

•  Contract  Language 

•  Software  Measurement 

»  Personnel  experience  or  training 

•  Ensure  implementation  and  adherence  to  processes  Software 
Measurement  for  Acquisition  of  Naval  Software  Intensive  Systems 

-  Guidebook  for  Software  Process  Improvement  for 
Acquisition  of  Naval  Software  Intensive  Systems 

»  Provide  support  to  acquisition  stakeholder  team 

•  Organize  to  capture  focus  teams  products 

•  Structure  follows  acquisition  process  timeline 
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Planning 


“Should-Be”  Software  Environment  & 

Bystems 
.  Engineer 


WBS 


SYSTEM 

1 

“SYS- 

Engr 

SW 

Engr 

1 

Logistics 

1 

rch 

to 


CSCI 

1 

CSCI 

2 

Build 

1 

Integration 

EailiiJaltfm 


oh  r  i 

i 

3DP 

wip 


4 - ► 


4 - ► 


tolbirks  (Cars  '6YJ) 


Management 


JjJda  Msmvmrs  « 

GATSSANDPt  \3BPS 


SW  Infused  WBS  Supports  Effective  Software  Metrics  and  Program  Management 


Institutionalization  Next  Steps  & 

1  Systems 

i  Engineer 

♦  Infuse  software  into  SE  Planning,  SE  Management,  and 
SE  Technical  Reviews  processes 

-  Systems  Engineering  Technical  Review  (SETR) 

-  Systems  Assurance 

-  Work  Breakdown  Structure  friendly  to  Software 

♦  Continue  working  with  USD(AT&L),  Services,  and  DAU 
to  meet  human  resources  and  training  needs 

♦  RDA  CHSENG  sponsor  next  updates  to: 

-  Software  development  techniques 

-  Contract  language  guidance,  when  required 
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Back-up  slides 


Infusion  Into  PoPS  for  Gate  Reviews 


IRda 

Chief 

Systems 

Engineer 


♦  Mapping  of  software  metrics-related  timeline  phases  to 
Gate  Reviews 


Lifecycle  Phases  SECNAVNOTE  5 


1:  Concept  Development 

Gate  1 

II:  Concept  Refinement 

Gates  2  &  3 

III:  Technology  Development 

Gates  4  &  5 

IV:  System  Development 

Gate  6 

V:  System  Demonstration 

Gate  6  (Phase  2) 

VI:  Production  &  Deployment 

Gate  6  (Phase  3) 

VII:  Operations  &  Support 

Gate  6  (Phase  4) 

-gcc  uaurxup  oiiuco  iui  uvci  vicw/ucouipuun  ui  caun  waic  rvcVIGW 

and  policy  memos  for  use  of  PoPS  methodology  at  Gate  Reviews 
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SPII  Core  Measurement 
and  Metrics  Update 


[Rda 

Chief 

Bystems 

Engineer 


Program  Office  Metrics 

Cause 


KPP  and  requirements 
driven 


Contract  Metrics 


Software  Size 


Effect 

KSLOC  and/or  function 
point  driven 


Cross  Functional  Match  -  Effective  Communications 


Key  Billets/Skills- 
DAIWIA  driven 


Organization 


Key  Billets/Skills - 
Contract/RFP  Identified 




rrr_ 

Cause 

Effect 

Government  Independent 
Cost  Estimate  (ICE);  Official 
Stamp  of  Program  Baseline; 
Delta  in  KPP/Requirements 

Cause 

KPP  and  requirements 
driven 

T&E  Outcomes 


Cost/Schedule 


Contract  Mods/Out  of 
Scope/Scope  Creep 
based  on  KPP/req  delta 


Software 

Quality 


Effect 

Defect  Rate/Cost  of 
Rework 

Based  on  Quality 


Based  on  KPP/Req  delta 


Details  are  dependent  on  SAM  organization  micro-product,  HR  skills  and  capability  micro-product;  Bl  contract  language  review 
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Motivation  for  SPII  Core  Metrics 


[Rda 

Chief 

Bystems 

Engineer 


Efforts  to  develop  appropriate  metrics  for  performance  measurement 

and  continual  process  improvement. 


Software  Size 


software  acquisition  planning 


Risk  Management 


requirements  development 

requirements  management 

ensure  that  key  program  personnel 
have  an  appropriate  level  of 
experience  or  training  in  software 
acquisition 


Project  Management  and 
Oversight 
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May  2006  SPII  Charter 


IRda 

Chief 

Systems 

Engineer 


“Successful  development  and  acquisition  of  software  is  paramount  for  acquiring 
Naval  Warfighting  and  business  systems.  There  are  many  parallel  and  related  efforts 
underway  that  address  improvement  in  the  acquisition  of  software  products: 
mandates  such  as  Public  Law  107-314  Section  804  and  the  Clinger-Cohen  Act; 
initiatives  such  as  Software  Assurance  and  Open  Architecture  (OA);  and  the 
development  of  best  practice  models  such  as  the  Capability  Maturity  Model 
Integration  (CMMI)  for  Acquisition.  To  consolidate  these  efforts  into  a  focused 
initiative,  I  have  formed  a  steering  group  composed  of  my  senior  engineering 
professionals  and  led  by  the  ASN  (RD&A)  Chief  Engineer.  This  group  will  evaluate 
existing  policies  and  implement  process  improvements  to  enhance  our  ability  to 
develop  and  acquire  software  without  sacrificing  the  cost,  schedule  and 
performance  goals  of  our  acquisition  programs. 

Additionally,  five  focus  teams,  led  by  department  software  engineering 
professionals,  have  been  established  to  achieve  our  strategic  software  goals  (see 
attachment): 

Software  Acquisition  Management  (SAM)  Focus  Team 
Software  Systems  Engineering  (SSE)  Focus  Team 
Software  Development  (SWDEV)  Techniques  Focus  Team 
Business  Implications  Focus  Team 
Human  Resources  Focus  Team” 
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ASN  RDA  Memo  dtd  May  15, 2006,  subj:  Software 
Process  Improvement  Initiative 


Business  Implications  (Bl) 


IRda 

Chief 

Systems 

Engineer 


♦  Accomplished  -  As  Is  and  To  Be 

-  Contract  Language  policy  memo  signed  17  Nov  2006 

•  Directs  standardized  contract  language  for  all  contracts  containing 
software  development,  acquisition  and  life  cycle  support  beginning 
with  RFPs  issued  after  1  Jan  2007 

-  Requires  developers  to  submit  Software  Development  Plan  (SDP) 

-  Contract  Language  Guidance  policy  memo  signed 
13  Jul  2007 

*  Provides  amplifying  guidance  information  on  the  17  Nov  2006 
Contract  Language  policy  memo 

♦  Institutionalize 

-  Re-enforced  in  the  overarching  Policy  and  Guidebook  for 
Acquisition  of  Naval  Software  Intensive  Systems  -  signed 
September  16, 2008 

-  Update  Contract  Language  based  on  future  need 
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Software  Development  Techniques 

(SWDT) 


IRda 

Chief 

Systems 

Engineer 


♦  Accomplished  -  As  Is  and  To  Be 

-  “Software  Development  Techniques  Phase  1  Report” 
signed  10  Jul  2007 

•  Provides  an  overview  of  existing  software  development  techniques 
and  suggestions  for  evaluating  emerging  software  development 
techniques 

•  Facilitates  program  managers  software  risk  management 

♦  Institutionalize 

-  Guidebook  for  Acquisition  of  Naval  Software  Intensive 
Systems  -  signed  September  16, 2008 

-  Annual  update  to  reflect  maturity  of  existing  techniques  and 
emergence  of  new  techniques 
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Software  Systems  Engineering  (SSE) 


IRda 

Chief 

Systems 

Engineer 


♦  Accomplished  -  As  Is  and  To  Be 

-  Program  Office  Survey  Findings  Report  promulgated  July 

*  Report  verifies  the  findings  of  previous  studies  (e.g.,  Defense 
Science  Board  (DSB)-200D  and  Naval  Research  Advisory  Committee 
(NRAC)-2006)  by  tracking  them  directly  to  current  programs  of 
record 

-  Software  Metrics  White  Paper  -  identified  4  core  metrics 

-  Develop  software  reviews  for  inclusion  in  Systems 
Engineering  Technical  Review  (SETR) 

♦  Institutionalize 

-  Software  Measurement  for  Naval  Software  Intensive 
Systems  Policy  -  signed  July  22, 2008 

•  Provides  a  set  of  software  metrics  to  assess  program  performance 

-  Incorporate  software  reviews  into  SETR  (planned  March 
2009) 

»  Executing  under  Systems  Engineering  Stakeholders  Group  (SESG) 
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Software  Acquisition  Management 

(MM) 


[Rda 

Chief 

Bystems 

Engineer 


♦  Accomplished  -  As  Is  and  To  Be 

-  “As  Is”  Report  signed  17  May  2007 

•  Uncovers  the  current  environment  for  the  acquisition  of  software  intensive 
systems  across  the  Naval  Enterprise 

•  Findings  are  consistent  with  past  DSB  and  NRAC  findings 

-  “To  Be”  Report  signed  6  Nov  2007 

•  Assists  acquisition  professionals  with  a  preview  of  key  considerations  for 
major  problems  having  been  found  to  be  most  troublesome  and  most 
commonly  documented 

♦  Institutionalize 

-  Tailorable  Organization  Structure  (included  in  Guidebook  Sept  2008) 

•  Tool  for  assessing  organizational  structure,  software  expertise,  and  staffing 
requirements  for  software  intensive  systems  program  offices 

-  Software  Measurement  for  Naval  Software  Intensive  Systems  Policy 
July  22, 2008 

•  Provides  a  set  of  software  metrics  to  assess  program  performance 

-  Use  the  Systems  Engineering  Plan  (SEP)  and  SETR  (planned  March 
2009) 

•  On-going  effort  through  the  SESG 
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Human  Resources  (HR) 


IRda 

Chief 

Systems 

Engineer 


♦  Accomplished  -  As  Is  and  To  Be 

-  “Role  Base  Right  Fit  Training”  Report  signed 
6  Nov  2007 

•  Addresses  the  training  issues  highlighted  by  the  SAM  focus  team 
“As  Is”  state  report,  SSE  focus  team  “Program  Management  Office 
Survey  Findings,”  DSB,  and  NRAC  findings 

♦  Institutionalize 

-  “Establishment  of  DAWIA  Software  Acquisition  Training 
and  Education  Working  Group”  draft  memo  by 
OUSD(AT&L) 

•  The  “Role  Base  Right  Fit  Training”  report  serves  as  Naval  input  to 
OSD  sponsored  reviews  of  software  acquisition  management 
competencies  for  six  acquisition  disciplines  (Program  Management, 
Contracting,  Acquisition  Logistics,  Systems  &  Software  Engineering, 
and  Legal) 
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Institutionalize  -  Guidebook 


IRda 

Chief 

Systems 

Engineer 


♦  Signatory:  ASN  RDA 

♦  Audience: 

-  Primary:  Government  acquisition  community 

-  Secondary:  Stakeholder  community  (e.g,  developers) 

♦  Objective: 

-  To  provide  support  to  acquisition  stakeholder  team 

-  Organize  to  capture  focus  teams  products 

-  Structure  follows  acquisition  process  timeline 

♦  Status:  Signed  September  16, 2008 
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Institutionalize  -  Policy  & 

J  Bystems 

^ Engineer 

♦  Signatory:  ASN  RDA 

♦  Audience: 

-  Primary:  Government  acquisition  community 

-  Secondary:  Stakeholder  community  (e.g,  developers) 

♦  Objective: 

-  Improve  software  acquisition  processes 

1.  Software  Measurement  for  Naval  Software  Intensive  Systems 

-  4  core  metrics 

2.  Overarching  Software  Process  Improvement  Policy  for 
Acquisition  of  Naval  Software  Intensive  Systems 

-  Software  Process  Management  Improvement 

-  Contract  Language 

-  Software  Measurement 

-  Personnel  experience  or  training 

-  Ensure  implementation  and  adherence  to  processes 

♦  Status:  signed  July  22, 2008  &  September  16, 2008 
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Weighting  of  Core  Metrics 


IRda 

Chief 

Systems 


Core 

Metric 

Gate  1  / 
Phi: 
Concept 
Development 

Gate  2/ 

Ph  II: 
Concept 
Refinement 

Gate  3  / 

Ph  II: 
Concept 
Refinement 

Gate  4  / 

Ph  III: 

Technology 

Development 

Gate  5  / 

Ph  III: 

Technology 

Development 

Gate  6  / 
PhIV: 
System 
Development 

Gate  6  Phase 
2/PhV: 
System 
Demon¬ 
stration 

Gate  6  Phase 
3 /Ph  VI: 
Production 
& 

Deployment 

Gate  6  Phase 
4/PhVII: 
Operations  & 
Support 

Size/ 

Stability 

[7 

10% 

10% 

10% 

20% 

30% 

25% 

30% 

30% 

30% 

Organ¬ 

ization 

[7 

50% 

40% 

50% 

40% 

30% 

25% 

15% 

15% 

15% 

Cost  / 
Schedule 

U 

30% 

40% 

30% 

25% 

25% 

25% 

30% 

30% 

30% 

Quality 

[7 

10% 

10% 

10% 

15% 

15% 

25% 

25% 

25% 

25% 

TOTAL 

100% 

c 

100% 

c 

100% 

€ 

100% 

c 

100% 

* 

100% 

e 

100% 

* 

100% 

0 

100% 

0 
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Software  Size/Stability  Metric 


IRda 

Chief 

Systems 

Engineer 


Phase 

I 

II 

III 

IV 

V 

VI 

VII 

Baseline/ 

Basis  of 

Metric 

Concept 
expectation 
of  %-age  of 
system 
functionality 
to  be 

delivered  by 
SW  (vice, 
e.g.,  HW) 

Concept 
expectation 
of  %-age  of 
system 
functionality 
to  be 

delivered  by 
SW  (vice, 
e.g.,  HW) 

SW  Size 
Estimates 

SW  Size 
Baseline 

SW  Stability 

SW  Stability 

SW  Stability 

Who  Collects 
Measure¬ 
ments 

Program 

Office 

Program 

Office 

Program 

Office  / 
Bidders 

SW 

developer/ 

integrator 

SW 

developer/ 

integrator 

SW 

developer/ 

integrator 

Program 

Office  / 

SW 

developer/ 

integrator 

Who  Analyzes 

Program 

Office 

Program 

Office 

Program 

Office 

Program 

Office  / 

SW 

developer/ 

integrator 

SW 

developer/ 

integrator 

SW 

developer/ 

integrator 

Program 

Office 

Metric 

%-age  of 
functionality 
in  SW 

%-age  of 
functionality 
in  SW 

Estimated 
SLOC,  FP, 
or  Req’ts. 

ESLOC,  FP, 
or  Req’ts. 

ESLOC,  FP, 
or  Req’ts. 

ESLOC,  FP, 
or  Req’ts. 

ESLOC,  FP, 
or  Req’ts. 

Use  of  Metrics 

Risk, 

Lessons 

Learned 

Risk, 

Lessons 

Learned, 

Concept 

Selection 

Risk, 

Lessons 

Learned, 

Source 

Selection 

Risk, 

Lessons 

Learned, 

Performance 

Risk, 

Lessons 

Learned, 

Performance 

Risk, 

Lessons, 

Learned, 

Performance 

Risk, 

Performance 
,  Lessons 
Learned, 
Database/ 
Archival 
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Software  Cost/Schedule  Metric 


IRda 

Chief 

Systems 

Engineer 


Phase 

I 

II 

III 

IV 

V 

VI 

VII 

Baseline/ 

SW  related 

SW  related 

Actual  SW 

Actual  SW 

Actual  SW 

Actual  SW 

Actual  SW 

Basis  of 

lERs,  SDXs 

lERs,  SDXs 

cost& 

cost  & 

cost  & 

cost  & 

cost  & 

Metric 

schedule 

schedule 

schedule 

schedule 

schedule 

data 

data 

data 

data 

data 

Who  Collects 

Sponsors  & 

Sponsors  & 

Program 

Program 

Program 

Program 

Program 

Measure¬ 

Advocates 

Advocates 

Office  /SW 

Office /SW 

Office /SW 

Office /SW 

Office/ 

ments 

developer/ 

developer/ 

developer/ 

developer/ 

SW 

integrator 

integrator 

integrator 

integrator 

developer/ 

integrator 

Who  Analyzes 

Sponsors  & 

Sponsors  & 

Program 

Program 

Program 

Program 

Program 

Advocates 

Advocates 

Office 

Office 

Office 

Office 

Office 

Metric 

# 

# 

Cost/Schedu 

Cost/Schedu 

Cost  / 

Cost / 

Cost/ 

lERs/SDXs 

lERs/SDXs 

le  Variance/ 

le  Variance/ 

Schedule 

Schedule 

Schedule 

produced  by 

produced  by 

Performance 

Performance 

Variance/ 

Variance/ 

Variance/ 

SW 

SW 

index 

index 

Performance 

Performance 

Performance 

index 

index 

index 

Use  of  Metrics 

Risk, 

Risk, 

Risk, 

Risk, 

Risk, 

Risk, 

Risk, 

Lessons 

Lessons 

Lessons 

Performance 

Performance 

Performance 

Performance 

Learned 

Learned 

Learned 

,  Lessons 

,  Lessons 

,  Lessons 

Lessons 

Learned 

Learned 

Learned 

Learned 
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Software  Quality  Metric 


IRda 

Chief 

Systems 

Engineer 


Phase 

I 

II 

III 

IV 

V 

VI 

VII 

Baseline/ 

Basis  of 

Metric 

SW  related 
IERS& 

SDXs 

SW  related 
IERS& 

SDXs 

Defects  per 
SLOC 

Defects  per 
SLOC, 

Defects  per 

system 

interface 

Defects  per 
SLOC, 

Defects  per 

system 

interface, 

Defects  per 

system 

interface 

Defects  per 
SLOC, 

Defects  per 

system 

interface, 

Defects  per 

system 

interface 

Defects  per 
SLOC, 

Defects  per 

system 

interface, 

Defects  per 

system 

interface 

Who  Collects 
Measure¬ 
ments 

Sponsors  & 
Advocates 

Sponsors  & 
Advocates 

Program 

Office/ 

SW 

developer/ 

integrator 

Program 

Office  / 

SW 

developer/ 

integrator 

Program 

Office/ 

SW 

developer/ 

integrator 

User/Tester 

User/Tester 

Who  Analyzes 

Sponsors  & 
Advocates 

Sponsors  & 
Advocates 

Program 

Office 

Program 

Office 

Program 

Office 

Program 

Office 

Program 

Office 

Metric 

%SW 

generated 

lERs/SDXs 

%SW 

generated 

lERs/SDXs 

Qty 

performance 

index/ 

Qty 

performance 

index/ 

Qty 

performance 

index/ 

Qty 

performance 

index/ 

Qty 

performance 

index/ 

variance 

variance 

variance 

variance 

variance 

Use  of  Metrics 

Risk, 

Lessons 

Learned 

Risk, 

Lessons 

Learned 

Risk, 

Lessons 

Learned 

Risk, 

Performance 
,  Lessons 
Learned 

Risk, 

Performance 
,  Lessons 
Learned 

Risk, 

Performance 
,  Lessons 
Learned 

Risk, 

Performance 
,  Lessons 
Learned 
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Software  Organization  Metric 


IRda 

Chief 

Systems 

Engineer 


Phase 

I 

II 

III 

IV 

V 

VI 

VII 

Baseline/ 

Basis  of 

Metric 

Effort/KSA 

Effort/KSA 

Effort/KSA/T 

urnover 

Effort/KSA/ 

Turnover 

Effort/KSA/ 

Turnover 

Effort/KSA/ 

Turnover 

Effort/KSA/ 

Turnover 

Who  Collects 
Measure¬ 
ments 

Program 

Office 

Program 

Office 

Program 

Office/ 

Bidders 

Program 

Office/ 

Contractor 

Program 

Office/ 

Contractor 

Program 

Office/ 

Contractor 

Program 

Office/ 

Contractor 

Who  Analyzes 

Program 

Office 

Program 

Office 

Program 

Office 

Program 

Office/ 

SW 

developer/ 

integrator 

Program 

Office/ 

SW 

developer/ 

integrator 

Program 

Office/ 

SW 

developer/ 

integrator 

Program 

Office/ 

SW 

developer/ 

integrator 

Metric 

Planned  #  of 
people  or 
planned  #  of 
labor  hours, 
KSA 

#  of  people 
or  #  of  labor 
hours/actual 
tmg  vs 
required  trng 

#  of  people 
or  #  of  labor 
hours/actual 
trng  vs 
required 
trng/#  of 
people  lost  & 
gained 

#  of  people 
or  #  of  labor 
hours/actual 
trng  vs 
required 
trng/#  of 
people  lost  & 
gained 

#  of  people 
or  #  of  labor 
hours/actual 
trng  vs 
required 
trng/#  of 
people  lost  & 
gained 

#  of  people 
or  #  of  labor 
hours/actual 
trng  vs 
required 
trng/#  of 
people  lost  & 
gained 

#  of  people 
or  #  of  labor 
hours/actual 
trng  vs 
required 
trng/#  of 
people  lost  & 
gained 

Use  of  Metrics 

Risk, 

Lessons 

Learned 

Risk, 

Lessons 

Learned 

Risk, 

Lessons 

Learned, 

Source 

Selection 

Risk, 

Lessons 

Learned 

Risk, 

Lessons 

Learned 

Risk, 

Lessons 

Learned 

Risk, 

Lessons 

Learned 
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ASM  (RDA) 
Chief  Engineer 


'mval  Pomr  ^Inhtgiiuor^'l^ 
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|\ 


Purpose 


Provide  an  information  briefing  on  the  ASN(RDA)  CHSENG 
initiative  to  improve  integration,  interoperability,  and  net- 
centricity  across  the  Department  of  the  Navy. 


2 


Agenda 


♦  Background 

♦  Overview  of  l&l  Management 

♦  Centralized  Planning  Processes 

♦  Decentralized  Execution  Processes 

♦  Capability  Package  Assessments 

♦  Configuration  Capture 

♦  Role  of  Integrated  Architectures 

♦  Governance  Structure 


3 


Background 


♦  In  February  2006,  ASN(RDA)  Chief  Systems  Engineer 
(CHSENG)  undertook  to  improve  systems  engineering 
across  the  department  in  the  area  of  integration  and 
interoperability  of  “information-handling”  systems. 

-  “Information-handling  system”  is  the  term  used  by  RDA  CHSENG  to 
cover  every  data  system  within  the  Department,  including  both  IT 
systems,  national  security  systems,  and  everything  else. 

♦  After  reviewing  the  existing  systems  engineering 
organizations  under  the  ASN(RDA),  CHSENG  determined 
that  the  best  value-added  for  the  CHSENG  was  to  accept  the 
role  of  systems-of-systems  engineer  at  the  Naval  mission 
level. 

-  PEO  systems  engineers  and  technical  directors  already  coordinated 
systems  engineering  within  their  organizations. 

-  PMO  system  engineers  held  responsibility  for  program-level 
systems  engineering. 
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Background 


But  a  gap  existed  at  the  echelon  above  where  any  PEO  had 
the  authority  to  operate  and,  as  a  result,  PEO-to-PEO 
collaboration  was  unsupervised  and  haphazard. 

-  ASN(RDA)  CHSENG  assumed  the  role  of  coordinator  for  issues 
which  cross  PEO  boundaries. 
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Background:  DoN  Systems  Engineering  Hierarchy 


6  6 


To  do  Enterprise  &  SoS  /  FoS  SE  need  to  Execute  Sound  System  SE  Practices 


Background 


♦  However,  to  establish  the  boundaries  within  which  the  RDA 
CHSENG  would  operate,  it  was  necessary  to  define  the  systems- 
of-systems  for  which  RDA  CHSENG  would  take  responsibility. 


-  We  created  the  DON  Enterprise 
Architecture  Hierarchy  to 
establish  those  boundaries. 

-  Aligns  Mission-Level  SOSs  to 
the  Joint  Capability  Areas. 

-  Resulting  mission-level 
architectures  will  describe  the 
Secretariat,  U.s.  Navy,  and  U.S. 
Marine  Corps’  contributions  to 
each  JCA. 

-  Approved  for  use  across  DON 
on  22  September  2008. 


Department  of  the  Navy 
Enterprise  Architecture  Hierarchy 
Version  1.0 


September  22. 2008 


Prepared  'ey 
Depanmeat  of  Navy 
Assistant  Secretary  of  the  Navy  I'FDA.i 
Chief  Svstems  Engineer 
ASN  RDA  CHSENG 
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Background 


♦  Sample  page  from  DON  EA  Hierarchy. 
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Integrated  Architectures 


Integrated  architectures  provide  the  means  for  defining  the 
details  of  the  operational  and  system  requirements. 

Integrated  architectures  are  needed  for  multiple  echelons: 

-  DON  Enterprise  Architecture. 

-  Mission-level  integrated  architectures  (244) 

-  Program/Systems:  ADNS,  AEGIS,  CVN,  LHA-6,  F/A-18 

Each  tier  of  integrated  architectures  as  a  subset  of  the  tier  above 
it. 
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Integrated  Architectures  (continued) 


♦  How  do  we  use  integrated  architectures? 


JCIDS  Rqmts  Naval  Architecture  Repository  System  Operational 

Developers  . _ _ _  *  Analysis  M&S  ! 


Program  SEs 


SOS 

Engineers 


Interfacing 

PMOs 
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Integrated  Hierarchic  Database 


Naval  Architecture  Elements 
Reference  Giude 


Analysis 

Technical 
Analysis  M&S 


CPA  Script  Library 


Architecture-Based  Models  Library 


Information  Support  Plans  and 
System/Program-Level 
Architectures 


-V- - 

Portfolio 

Mgrs 


System 

OPEVAL 


o 


3Z 


DON 

Secretariat 


Pre-Deployment 

Capability 

Package 

Assessments 
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Overview  of  l&l  Management 


♦  First  order  of  business  was  to  identify  ALL  of  the  missions  in  the 
Department  of  the  Navy  (DON). 

-  Requires  a  definition  of  a  Naval  mission. 

♦  Naval  missions  are  defined  as  the  Navy,  Marine  Corps,  and 
Secretariat  contributions  to  the  Joint  Capability  Areas  (JCAs). 

-  Results  in  244  mission  areas,  based  on  2007  JCAs. 

-  These  are  listed  and  collated  in  the  DON  Enterprise  Architecture 
Hierarchy. 

-  Will  be  updated  following  revisions  to  the  JCAs  scheduled  for  November 
2008. 
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Overview  of  l&l  Management  (continued) 


♦  Because  of  the  complexity  of  the  Department  of  the  Navy  (DON), 
RDA  CHSENG  relies  on  assistance  provided  by  Mission-Area 
Chief  Engineers  who  are  experts  in  particular  systems-of- 
systems  and/or  mission  areas. 

-  FORCEnet:  SPSWARSYSCOM  5.1 

-  Sea  Shield:  NAVSEASYSCOM  05W 

-  Sea  Strike/Shaping  (Air,  Sea,  Land,  INFO  OPS,  SPECWAR) 

-  Sea  Basing:  To  be  determined. 

-  Expeditionary  Maneuver  Warfare  (MARCORSYSCOM  DEP  for  ENG) 

-  Manpower,  Personnel,  Training,  Education:  To  be  determined. 

-  Sea  Enterprise:  To  be  determined. 
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Overview  of  l&l  Management  (continued) 


♦  We  are  implementing  an  end-to-end  management  process  for  l&l 
of  information  systems  which  is  based  on  the  systems 
engineering  needed  by  the  mission-level  system-of-systems. 

♦  Uses  a  philosophy  of  Centralized  Planning  -  Decentralized 
Execution  -  Independent  Assessments  -  Configuration  Capture. 

♦  Relies  on  multi-tiered  integrated  architectures  to  set  technical 
requirements  and  to  communicate  among  engineers. 
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Centralized  Planning 

♦  Objectives  for  Centralized  Planning  include: 

-  Consistent  application  of  standards  across  PEOs/SYSCOMs. 

-  Ensuring  full  understanding  of  the  role  of  a  single  system  within  the  SoSs 
where  it  participates.  Overseeing  the  resolution  of  issues  among 
PEOs/SYSCOMs. 

-  Conduct  initial  evaluations  of  the  operational  effectiveness  and  technical 
performance  of  the  mission-level  SoSs. 

♦  The  Information  Support  Plan  provides  the  means  for 
accomplishing  Centralized  Planning  across  PEOs/SYSCOMs 
and  with  higher  authorities. 

-  Reviewed  at  each  acquisition  milestone  and  each  major  upgrade. 


Centralized  Planning  Methods: 

♦  Establishment  of  system-level  and  mission-level  integrated 
architectures. 

♦  Comparison  of  architectures  of  new  systems  with  mission 
architectural  baselines. 

♦  Review  of  other  ISP  and  NR-KPP  requirements. 

♦  Concurrence  from  PMOs  of  interfacing  systems. 

♦  Concurrence  from  CIO/DCIO(N)/DCIO(MC). 

♦  Concurrence  from  NNWC,  MCCDC  and  operational  agents. 

♦  Use  existing  processes  for  reviews  of  ISPs. 

-  DON-level  review. 

-  DOD-level  review  using  JCPAT-E 


De-Central  ized  Execution 


♦  PMs  and  PEOs  execute  their  acquisition  programs  according  to 
plans  (SEP,  ISP). 

♦  ASN(RDA)  CHENG,  coordinating  with  the  DON  Engineering 
community,  assists  by: 

-  Providing  a  venue  for  coordinating  across  PEOs,  especially  to  resolve  cross- 
PEO/SYSCOM  issues, 

-  Providing  common  dictionaries, 

-  Developing  and  distributing  mission-level  integrated  architectures. 

-  Developing  and  interpreting  policies  of  higher  headquarters, 

-  Supporting  program  representation  to  higher  headquarters, 

-  Providing  a  communications  link  to  authoritative  sources  within  the 


operational  agents. 


De-Centralized  Execution  (continued) 


♦  Revised  ISPs  and  system-level  DT/OT  test  reports  provide  the 
means  for  oversight  of  De-Centralized  Execution. 


Independent 

Assessment 


Independent  Assessments 


♦  There  is  a  need  for  formal  evaluation  of  the  performance  of 
mission-level  systems-of-systems. 

-  OPEVAL  concentrates  on  single  systems  only. 

-  Evaluation  needs  to  be  done  in  an  operationally-relevant  context. 

♦  Capability  Package  Assessments  (CPAs)  will  become  the  means 
for  independent  testing  of  SOSs. 

-  Based  on  a  process  prototyped  by  MCSC/MCTSSA  since  FY02. 

-  Aligns  with  NNWC  desire  for  more  relevant  SOS  assessments. 

♦  Evaluation  criteria  are  defined  by  the  mission-level  integrated 
architecture. 


Independent  Assessments  (continued) 

♦  Test  scripts  are  developed  for  CPAs  from  the  following  MCP- 
level  architectural  views: 

-  OV-5  Activity  Model, 

-  OV-6C  Operational  Event  Trace  Description, 

-  SV-1/2  Systems  Interface  and  Communications  Description, 

-  SV-5  Operational  Activity  to  Systems  Function  Matrix, 

-  SV-1 OC  Systems  Event  T race  Description 

♦  Initial  test  thread  is  Close  Air  Support. 

♦  We  are  coordinating  with  NNWC  for  access  to  conduct  CPAs 
during  battle  group  pre-deployment  work-ups. 
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Configuration  Capture 


The  configuration  observed  aboard  the  battlegroup  during  the 
CPAs  will  be  incorporated  into  the  architecture  repository  as  the 
“As-ls”  configuration  for  the  afloat  portion  of  the  DON  Enterprise 
Architecture. 

-  CPA  configurations  and  results  inform  the  mission-level  integrated 
architectures  of  real-world  conditions. 


OPEVAL 

Centralized  _ _  ISP _ __  De-Centralized  _ 

RpMe-gJ 

Independent 

Planning  Execution 

ISP 

Assessment 

INTEGRATED 

ARCHITECTURE 


Configuration  Capture 


DGSIT  CPA 
Report 
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Navy  Component 
Commander  (COCOM) 


USMC  Component 
Commander  (COCOM) 


JTFHQ 


JFACC 


JFMCC 

.1 

OTC 

JFLCC 


CVTG: 


CVN(s) 

CG-47 

DDG-51 

SSN 


V 


Not  Shown: 
MNW,  LSG, 
Sea  shield 
functions. 


CAS  Aircraft: 


POSITION/LOCATION 

^SOURCE 


CAS 

AIRCRAFT 

4*- 


ARTILLERY 

BATTERY 

FDC 


GIG  and  FORCEnet  Systems/Services: 
Comms  &  Networking  Infostructure 
C2/DS  Systems 
ISR/BA  Systems 


TDN/WIN-T  Systems 
MAGTF  C2  Systems 


ZJJ 


MA  CHENGs 


l&l  Management  Structure 


NNFE 


Mission 
Architecture 
Support  Team 


NR-KPP 

NSWG 

Aggregation 

Process 

Product 

Product 

Team 

Team 

Team  (FY09) 
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Agenda 


•  MS2  Tactical  Systems 

•  Motivation  for  Survivabteffletworks:  C4ISR 

•  A  Framework  for  cost-effi^^m*  survivable  network 
design 

•  Summary/Discussion 


NDIA  Conference  Oct  20,  2008 
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MS2  Tactical  Systems  - 
C4ISR  Products  and  Solutions 


MS2  Tactical  Systems  Delivers  and  Supports  Complex  C4ISR  Solutions 


NDIA  Conference  Oct  20,  2008 


©2008  Lockheed  Martin 


Motivation:  Complexity  of  C4ISR  and  Battle  - 

Management 

•  Sensors:  They  are  everywhere  on  many  networks 

-  Lots  of  data  in  many  types  and  formats 

-  Diverse  capabilities:  range ,  modality,  maneuverability 

-  Networks  are  poorly  integrated 

•  Communications  and  disserrtination 

-  Inter  and  intra  networking 

-  Networking  platforms  have  different  characteristics:  mobility,  power, 
line-of-sight,  latency,  bandwidth 

-  Network-to-network  adaptation:  adaptive  data  rate  and  waveforms 

•  “ Always-on  Connectivity  anytime,  anywhere,  anyhow 


Objective:  Reliable  information  transfer  under  dynamic  conditions  with  QoS 
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What  is  a  Survivable  Network? 


A  survivable  network  has  the  characteristic  that 
essential  services  are  preserved  under  disruption  and 
recover  full  services  tJma  timely  manner 

•  Disruption  can  resun  ft^mtljm^mdctors 

-  Congestion  resulting  from  excess  offered  load 

-  Protocol  Interworking  failure  (configuration) 

-  Physical  disruption 

-  Security  failure  (Denial  of  service) 

•  Service  recovery 

-  Priority  of  restoral 

-  Automated  vs  manual 

-  Efficiency  (recover  full  service  in  a  timely  manner) 
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Survivability  Framework:  Three  levels  of  Networl 
Integrity  during  undesirable  events 


•  Network  availability  (planned) 

-  Normally  associated  with  maintenance 
and  configuration  faults  (single  fault) 

-  Represents  the  majority  of  faults  V 

r  J  J  >-  Logical  Layer  recovery 

-  Automated  recovery  or  inherent  reliability  f  *n  (Application  and  traffic  layer) 

in  the  design 

•  Single,  worst  case  failure  (node, 

link,  etc)  J 

-  Environmental  failure 

-  Accident  V  Du  ■ 

>■  Physical  Layer  recovery 

-  Manual  recovery  (minutes/hours)  f 

•  Disaster-based  event:  Several 
links  or  nodes  fail  simultaneously 

-  Natural  or  man-made  event 

-  Manual  recovery  (lengthy-  > 

hours/days/weeks)  ^ 
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Network  Level  Emergent  Behavior:  System  View 


•  System  Requirements  need  to  be  integrated  with 
survivability  requirements  atmode  and  network  level 

-  Organize  into  essential  and  non-essential  services 

-  Organize  by  user  orlbusiness  function 

•  Survivability  imposes  nei/Uffopes  of  requirements 

-  Emergent  behavior:  collective  behavior  of  node  services 
communicating  across  the  network 

-  Adaptive  behavior ,  function ,  and  resource  allocation 


Example:  Functions  ana  resources  devoted  to  non- 
essential  services  could  be  reallocated  to  essential 
services 
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Sample  Survivability  Measures 


•  Connectivity  based  measures 

-  Route  availability  ratio 

-  Probability  of  node  isolation 

•  Traffic  based  measures 

-  Average  network  blocking  given  a  failure 

-  Average  number  of  lost  calls  given  a  failure 

•  Desirable  characteristics  of  measures 

-  Technology  independent 

-  Measure  survivability  under  the  three  described  levels  of  failure 

-  Can  be  applied  to  a  subnetwork  of  the  network 

-  Can  measure  the  customer/user  impact 
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Survivability  Framework:  Analysis 


Level  1  Level  2 

Network  Availability  Single,  Worst  Case  Event 

Performance 

(Fault  Tolerance  + 

Security) 

Availability 

(System  Reliability) 


Recovery  Time 

(Modifiability) 


Level  3 

Disaster-based  Event 
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Optimization  Techniques 


•  Architectural  trade  analysis  using  design  patterns  and 
styles 

-  DoDAF  modeling 

-  Exhibit  300 

•  Formal  methods  uMnd  Markov  modeling  and 
simulation 

-  Hamiltonian  Cycle  based  analysis 

-  Generalized  graph  methods  for  clustering 

-  Minimum-cost  vertex-connectivity  analysis 

•  Scenario  based  methods 
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Service  Recovery  and  Efficiency 


•  Maintainability:  Fewer  uniquejnstallations 

-  Default  configurations 

-  Training 

-  Logistic  support 

•  Operational  availability 

-  Faster  restoral 

-  Swap  like  components 

-  Priorities:  Know  when  I  need  a  service 

•  Life  cycle  cost  management 


Objective:  Commonality  across  the  Enterprise 
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Summary 


•  The  emphasis  on  net-centric  operations  makes  it 
essential  that  we  create?  effective  methods  for 
survivable  network  design 

•  We  can  apply  systermenpifj^brincj  methodologies 
similar  to  those  we  appftfMmmner  systems  in  order  to 
define  “essential”  services 

•  We  can  use  spiral  modeHSTanalysis  and  design  with 
appropriate  measures  to  obtain  desired  properties 
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Questions? 


Den  .  cb  m 

(651)456-2421 
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Agenda 

»  Why  are  good  requirements  so  important? 
»  What  makes  a  good  requirement? 

»  Requirements  definition 
»  Managing  change 
»  Advantages  in  modeling 
»  Effective  prototyping 
»  Summary 
»  Q  &  A 
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Why  Are  Good  Requirements  So 
Important? 


Change  Impact  vs.  Project  Phase 


Project  Phase 


Why  Are  Good  Requirements  So 
Important?  (cont) 

»  Requirements  can  be: 

» Unrealistic 
» Incomplete 
» Ambiguous 
» Contradictory 
»  Un-testable 
»  Poorly  managed 

»  This  leads  to: 

» Rework,  delays,  budget  over-runs,  unhappy 
customers 
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What  Makes  a  Good  Requirement? 

»  Be  S.M.A.R.T.* 

» Specific  (concise,  clear,  unique) 

» Measurable 
» Achievable 
» Relevant 
» Testable 

»  What  vs.  How 
»  This  leads  to: 

»  Less  rework,  shorter  schedules,  lower  costs,  happy 
customers 

»  *http://www.win.tue.nl/~wstomv/edu/2ip30/references/smart-requirements.pdf 
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Requirements  Definition 

»  Consider  interests  of  ALL  stakeholders 
»  Include  all  users  in  reviews 

»  End  user 

»  Development/Safety  Team 
»  Production/Maintenance  Team 
»  VerificationA/alidation  Team 

»  Don’t  forget: 

»  Traceability 
»  Interface  requirements 
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Requirement  Layers 

»  Start  with  high  level  concept  and  technical 
requirements 

»  Drill  down  adding  more  detail  with  each  layer 

» Highest  level  -  capabilities 

» Next  n  levels  -  subsystems,  architecture,  high  level 
design,  low  level  design 

» The  number  is  subjective  -  depends  on  complexity 

» Stop  when  you  have  enough  detail  to  build  it,  buy  it, 
code  it,  and  test  it 
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Traceable 


Requirements  Layers 


Customer  Requirements 
System  Requirements 
Subsystem  Requirements 

Component/Part  (H/W  &  S/W )  Req. 
Verif./Valid.  Procedures 
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Managing  Change 

»  During  initiation: 

»  Define  and  formalize  change  control  process  (internal 
and  external) 

» Define  how  legacy  issues  will  be  handled 

»  Get  to  know  the  “customer”  and  learn  their  true 
priorities 

»  Good  communications  with  stakeholders  is  key 
(include  Contract  Administrators) 
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Managing  Change  (cont.) 

»  Effectively  and  formally  evaluate  and  control 
proposed  changes 

»  Hold  the  line  even  on  small  impact  changes 

»  Requirements  vs.  desirements  (what  is  in  the 
contract?) 

»  Identify  and  address  errors/issues  as  early  as 
possible 
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Advantages  of  Model  Based 
Development 

»  Early  detection  of  errors  in  requirements  and 
design 

»  Proof  of  concept 
»  Repeatable 

»  Reduces  impact  of  changes 

»  Reduces  cost  of  downstream  activities  (design, 
code) 
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Rapid  Prototyping 

»  Formalize  the  process  to  provide  proof  of 
concept 

»  Make  it  repeatable  -  what  if  it  works? 

»  Emphasis  of  testing  on  core  functionality, 
doesn’t  address  capabilities  such  as  operational 
environment 
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Summary 

»  Create  S.M.A.R.T.  requirements 

»  Communicate  with  stakeholders  and  dig  deeper 
for  clarification  of  requirements 

»  Formalize  the  change  management  process 

»  Identify  legacy  issues  at  the  start  of  the  project 

»  Leverage  modeling  to  detect  errors  early  and 
reduce  downstream  costs 

»  Use  prototyping  to  help  test  functionality 
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Questions? 


Contact  Info: 

Scott  Derby 
Programs  Manager 
Esterline  AVISTA 
Phone  (608)348-8815 
Fax  (608)348-8819 
Email  Scott.Derbv@avistainc.com 
www.avistainc.com 
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Timothy  G.  Olson,  President 
Lean  Solutions  Institute,  Inc. 
(760)  804-1405  (Office) 
Tim.Olson@lsi-inc.com 

www.lsi-inc.com 
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World-Class  Quality 


“I  have  made  this  letter 
longer  than  usual 
because  I  lack  the  time 
to  make  it  shorter” 

Blaise  Pascal 


Training  Material  Used  with  Permission  and  Licensed  by  Lean  Solutions  Institute,  Inc.  (LSI) 
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World-Class  Quality _ 

Objectives 

Provide  some  “lean  results”  motivation. 

Describe  some  engineering  problems  from 
industry. 

Describe  motivation  and  advantages  of 
architectures. 

Describe  motivation  and  advantages  of  models. 
Provide  some  examples. 

Answer  any  of  your  questions. 
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Outline 


World-Class  Quality 


Lean  Results 


Some  Problems  in  Engineering 
Systems  Engineering  Processes 
Why  Focus  on  Architectures? 

Why  Focus  on  Models? 

The  Future:  Industry  Standards  and  Tools 
Summary 
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The  Quality  Crisis 

The  cost  of  poor  quality: 

•  “In  most  companies  the  costs  of  poor  quality 
run  at  20  to  40  percent...  In  other  words, 
about  20  to  40  percent  of  the  companies’ 
efforts  are  spent  in  redoing  things  that  went 
wrong  because  of  poor  quality”  (Juran  on 
Planning  for  Quality,  1988,  pg.  1) 

•  Crosby’s  Quality  Management  Maturity  Grid 
states  that  if  an  organization  doesn’t  know 
it’s  cost  of  quality,  it’s  probably  at  least  20%. 
(Crosby,  Quality  is  Free,  1979,  pg.  38-39) 
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World-Class  Quality 

What  is  Lean? 

Lean  has  its  roots  in  quality  and  manufacturing, 
and  is  a  recent  popular  movement  in  quality. 

“Lean  Production”  is  the  name  for  the  Toyota 
Lean  Production  System. 

The  following  are  major  lean  references  (see 
references  in  back  of  presentation  for  full 
references): 

•  “The  Machine  That  Changed  The  World” 

•  “Learning  to  See” 

•  “The  Toyota  Way” 

•  “The  Toyota  Product  Development  System” 

•  “Lean  Thinking” 
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Some  Lean  Principles  -  (1) 

Establish  customer  defined  value  (i.e.,  identify 
the  “value  stream”).  Process  =  “value”. 

Continuously  eliminate  non-value  added 
activities  (e.g.,  waste,  rework,  defects). 

Use  leadership  and  standardization  to  create  a 
lean  culture. 

Align  your  organization  through  visual 
communication. 

Create  an  optimized  process  flow  (e.g.,  “Flow”, 
“Pull”,  “Just-In-Time”,  “Leveled”). 
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gome  Lean  Principles  -  (2) 

Use  lean  metrics  to  manage  the  value  stream. 

Front-Load  the  process  for  maximum  design 
space. 

Build  a  learning  organization  to  achieve  lean 
and  continuous  improvement. 

Adapt  technology  to  fit  your  people  and 
processes. 

Strive  for  perfection  through  continuous 
improvement. 
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Some  Lean  Results 


MEASUREMENT 

WORLD-CLASS  BENCHMARK 

Costs  of  Poor  Quality 
(COPQ) 

Reduced  from  ~33%  to  ~15% 

(e.g.,  cut  COPQ  in  half) 

Defect  Removal  Efficiency 

70-90%  defect  removal  before  test 

Post-Release  Defect  Rate 

Six  Sigma  (i.e.,  3.4  Defects  Per  Million) 

Productivity 

Doubled  (e.g.,  in  5  years  at  ~20%  a  year) 

Return  on  Investment 

7:1  -12:1  ROI 

Schedule  /  Cycle  Time 

Reduced  by  10-15%  (e.g.,  per  year) 
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Some  Engineering  Problems 

Numerous  problems  with  requirements. 

Too  many  defects  (i.e.,  quality  problems). 

Lack  of  metrics  (e.g.,  process  improvement). 

Major  decisions  made  made  subjectively  or 
without  data. 

Management  problems  (e.g.,  poor  risk 
management). 

Lack  of  product  integrity. 
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Example  Problem:  Requirements 

A  research  report  from  the  Standish  Group 
highlighted  the  continuing  quality  and  delivery 
problems  in  our  industry  and  identified  three 
leading  causes: 

*  Lack  of  user  input 

*  Incomplete  requirements  and  specifications 

*  Changing  requirement  specifications 


•  Reference:  “Chaos”,  Compass,  The  Standish  Group,  1997,  used  with  permission. 
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^  Problems  with  Requirements 

According  to  the  SEI  [Christel  92],  problems  of 
requirements  elicitation  can  be  grouped  into  3 
categories: 

1.  Problems  of  Scope:  the  requirements  may 
address  too  little  or  too  much  information. 

2.  Problems  of  Understanding:  problems  within 
groups  as  well  as  between  groups  such  as 
users  and  developers. 

3.  Problems  of  Volatility:  the  changing  nature 
of  requirements. 
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CMMI®  Process  Areas 

Engineering: 

*  Requirements  Management  (REQM) 

*  Requirements  Development  (RD) 

*  Technical  Solution  (TS) 

*  Product  Integration  (PI) 

*  Verification  (VER) 

*  Validation  (VAL) 

Support: 

*  Measurement  and  Analysis  (MA) 

*  Process  &  Product  Quality  Assurance  (PPQA) 

*  Configuration  Management  (CM) 

*  Decision  Analysis  and  Resolution  (DAR) 

*  Causal  Analysis  and  Resolution  (CAR) 

®  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University 
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CM  Ml®  Process  Areas 

Project  Management: 

*  Project  Planning  (PP) 

*  Project  Monitoring  and  Control  (PMC) 

*  Supplier  Agreement  Management  (SAM) 

*  Risk  Management  (RSKM) 

*  Integrated  Project  Management  +  IPPD  (IPM) 

*  Quantitative  Project  Management  (QPM) 

Process  Management: 

*  Organizational  Process  Definition  +  IPPD  (OPD) 

*  Organizational  Process  Focus  (OPF) 

*  Organizational  Training  (OT) 

*  Organizational  Process  Performance  (OPP) 

*  Organizational  Innovation  and  Deployment  (OID) 

®  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University 
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NASA  Systems 
Requirements 


Engineering 

(NPR-7123) 


Requirements  Flow  Down 
to  Level  below 


Realized  Products 
from  Level  below 


System  Design  Processes 
applied  to  each  WBS  Model 
down  and  across 
system  structure 


Product  Realization  Processes 
applied  to  each  product 
up  and  across 
system  structure 
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^  Why  Architectures? 

Architectures  are  very  powerful  because  they: 

•  Are  graphical  (a  picture  is  worth  a  1000  words) 
and  can  be  powerful  communication  tools. 

*  Provide  a  framework  for  how  components  are 
related  (e.g.,  interfaces,  interdependencies, 
relationships)  and  how  components  fit  together. 

•  Promote  reuse  (e.g.,  products,  components, 
requirements,  designs,  tests,  interfaces,  etc.) 
and  can  improve  productivity  and  quality. 

*  Can  be  modeled  in  an  automated  tool. 
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Architectures  consist  of: 

*  Components 

*  Interfaces,  interdependencies,  and  other 

relationships  among  components 

*  Ordering  and  rules  for  putting  components 
together 

Simple  Architecture  Example:  Lego’s 

Numerous  Types  of  Architectures: 

*  Product  and  Domain  Specific  Architectures 

*  Business,  Data,  Technology,  etc.  Architectures 

*  Discipline  Specific  Architectures  (e.g.,  software) 

*  Process  Architectures 

*  Documentation  Architectures 
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Example  Product  Architecture 
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Example  Process  Architecture 


Management  Processes 


Project 

Management 

Risk 

Management 

Supplier 

Management 

Engineering  Processes 

Requirements 

Design 

Implementation 

Test 

Support  Processes 

Configuration 

Management 

Auditing 

Measurement 
and  Analysis 

Decision 
Analysis  & 
Resolution 
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Documentation  Architecture 


•  Slide  adapted  from”A  Software  Process  Framework  for  the  SEI  Capability  Maturity  Model”,  CMU/SEI-94-HB-01 
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w'  Why  Models? 

Models  are  very  powerful  because  they: 

•  Are  graphical  (a  picture  is  worth  a  1000  words) 
and  can  be  powerful  communication  tools. 

•  Can  scale  up  to  complex  systems  and  provide  a 
tool  to  analyze  complex  relationships  and 
dependencies. 

•  Promote  reuse  (e.g.,  products,  components, 
requirements,  designs,  tests,  interfaces,  etc)  and 
can  improve  productivity  and  quality. 

•  Can  be  represented  in  an  automated  tool,  and 
simulated. 
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w'  Models 

Models  are  abstractions  of  reality  constructed  for  a 

(useful)  purpose  consisting  of: 

•  Formal  notations  and  rules  for  representations 

•  Model  components  or  building  blocks 

•  Ways  to  model  interfaces,  interdependencies, 
and  other  relationships  among  the  model 
components 

There  are  numerous  modeling  languages  and  tools. 

A  Few  Modeling  Examples: 

•  Behavioral  Models  (e.g.,  timing,  states) 

•  Structural  Models  (e.g.,  hierarchy,  order) 

•  Functional  Models  (e.g.,  input,  function,  output) 

•  Process  Models  (e.g.,  the  5  W’s) 
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What  is  a  Process  Model? 

Process  Model: 

•  An  abstraction  of  a  process  typically  characterized 
by  formal  notations  for  representing  roles, 
activities,  and/or  work  products,  and  the 
relationships  (e.g.,  events,  transformations)  among 
them. 

Types  of  process  models: 

•  Descriptive  (as-is):  describes  what  is  actually  done 

*  Prescriptive  (to-be):  prescribes  what  to  do  (e.g.,  by 
new  policies,  standards,  process  guidelines,  etc.) 

*  Mixed  (both):  most  process  models  are  a  mixture  of 
prescriptive  and  descriptive  processes 
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Popular  Process  Models:  CMM/CMMI 

SADT:  Structured  Analysis  and  Design  Technique 
(SADT)  is  a  graphical  systems  modeling  language 
developed  at  Softech/MIT  by  Doug  Ross  in  early 
1970's.  Used  extensively  to  document  all  manner  of 
systems  including  manufacturing  processes.  Has 
automated  tool  support  (e.g.,  IDEF). 

ETVX:  Entry  criteria/Tasks/Verification/eXit  criteria 
(ETVX).  Developed  at  IBM  in  the  mid  1980's.  Simple  to 
use,  but  no  automated  tool  support. 

Role/Flow  or  Swim-Lane  Models:  Like  flow  charts,  but 
have  swim-lanes  for  roles  and  are  formal  process 
models.  Have  become  very  popular  in  the  last 
decade.  [Example  Handout]. 
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Example  Requirements  Process: 
NASA  Onboard  Shuttle  Project 
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State  of  the  Industry  - 
Process  Management 


Three-Ring  Binders 

•  Demonstrated 
organization  commitment 

•  Often  became  shelfware 


Websites 

•  More  accessible  by 
practitioners 

•  Often  difficult  to  navigate 
and  maintain 


Model-Driven 

•  Access  to  industry 
standard  frameworks 

•  Integration  of  multiple 
lifecycles 

•  Formal  process  asset 
management 


Copyright  ©  1998-2008,  Armstrong  Process  Group,  Inc.,  All  rights  reserved. 
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^  Benefits  of  a 

Standards-Based  Approach 

Increased  sustainability: 

*  Lower  cost  and  shorter  time  of  initial  adoption 

*  Widespread  availability  of  knowledgeable 
employee,  contractor,  and  vendors 

*  Lower  cost  of  maintenance 

Lower  risk: 

*  Apply  proven  best  practices 

*  Widespread  adoption  across  industry 


Copyright  ©  1998-2008,  Armstrong  Process  Group,  Inc.,  All  rights  reserved. 
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Some  Industry  Standards 

OMG:  Object  Management  Group 

UML:  Unified  Modeling  Language 

SvsML:  Systems  Modeling  Language 

SPEM:  Software  and  Systems  Process  Engineering 
Metamodel 

Eclipse  Process  Framework  (EPF)  Composer:  Open 
source  based  on  SHbM:  www.eciipse.org/ept. 

OpenUP:  Open  Unified  Process  -  process  framework 

TOGAF:  The  Open  Group  Architecture  Framework 

DoDAF:  DoD  Architecture  Framework 

IEEE  1471:  Recommended  Practice  for  Architecture 
Description  of  Software  Intensive  Systems 
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Summary 

There  are  many  industry  engineering  problems. 

Systems  engineering  needs  to  focus  on  improving 
those  engineering  problems. 

Organizations  need  lean  measurable  results  (e.g., 
7:1  ROI). 

Architectures  and  models  are  powerful  tools  to 
help  improve  engineering  and  obtain  measurable 
results. 

The  future  of  architectures  and  models  is  industry 
standards  and  tools.  Architectures  can  also  be 
represented  with  models. 
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Architecture  and 
Model  Based 
Systems  Engineering 
For  Lean  Results 


NDIA  Systems  Engineering  Conference  -  October  2008 


Timothy  G.  Olson,  President 
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(760)  804-1405  (Office) 
Tim.Olson@lsi-inc.com 
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Stryker  F  amily  of  V  ehicles 


Mobile  Gun  System  (MGS)  27 


NBC  Reconnaissance  Vehicle 
(NBCRV)  3 


Engineer  Squad  Vehicle 
(ESV)  13 


120mm  Mounted  Mortar  Carrier 
(MC-B)  37 

Anti  Tank  Guided  Missile 
(ATGM)  10 


Infantry  Carrier  Vehicle 
(ICV)  130 


Commander’s  Vehicle 
(CV)  28 


Fire  Support  Vehicle 
(FSV)  14 


Reconnaissance  Vehicle 
52 


Medical  Evacuation  Vehi 
(MEV)  16 
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BLUF  -  Key  Factors  for  Successful 
Reliability  Growth  Program 

•  Program  Management  -  Integrated  Team 

71  The  systems,  tools,  and  practices  now  in  place  between  the  US  Government  and  General 
Dynamics  Land  Systems  allowed  the  system’s  reliability  to  grow  (repeatable  process) 

71  Reliability  growth  requires  commitments  from  Material  Developer  Team,  Combat 

Developer,  and  Independent  Test  and  Evaluation  Communities  (requirements,  test,  data, 
methodology,  tools) 

•  System  Engineering  -  Reliability  Backbone 

71  Integrates  All  Reliability  Tasks 

71  Redirects  Tasks  Toward  a  Single  Objective 

71  Crosses  Boundaries  Affecting  Operational  Reliability 

71  Provides  Program  Manager  Authority,  Funding,  and  Focus  on  Engineering,  Processes, 
Documentation,  Training,  Manufacturing,  and  Testing  for  Reliability 

•  Reliability  Data  Analysis  -  Reliability  Assessment 

71  FDSC  -  Failure  Definition  Scoring  Criteria 
71  Failure  Categories 

■  Inherent  vs.  Induced  Reliability 
71  Mission  Profile  and  Life  Variable 
71  Data  Grouping  and  Modeling 
7i  Instantaneous  vs.  Cumulative  Reliability 
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MGS  Program  Management 


Plan 


•  Phase  I  -  Conduct  an  Additional 
Reliability  Test  (ART) 

7i  Validate  effectiveness  of  216  PQT  and 
Post-PQT  corrective  actions 


•  Phase  II  -  Implement  changes  to 
Government  and  GDLS  Systems 
Engineering  Processes 
7i  Management  and  process  changes 


•  Phase  III  -  Redesign  of  Sub-System 
components  and  integration 


Tests 

•  Additional  Reliability  Testing  (DEC  2004 
-  MAR  2005) 

7i  2  vehicles 

71  Pre-ART  -  XXX  rounds  &  X00  miles 
71  ART  -  XXX  rounds  &  X,000  miles 
7i  Reliability  Point  Estimate  XX  MRBSA 


•  Reliability  Growth  Test  (JUL-AUG  2005) 

7i  2  Vehicles 
71  XXX  rounds 
71  X,000  miles 

71  Reliability  Point  Estimate  XX  MRBSA 

•  Production  Verification  Testing  (APR 
2006  -  DEC  2007) 

7i  3  Vehicles 
7i  XXXX  rounds 
71  XX, 000  miles 

7i  On-going  -  Current  estimate  XXX  MRBSA 
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Agenda 


•  What  is  MGS 


•  Success  Factors  of  MGS  PVT 

71  Program  Management  -  Integrated  Team 

■  FDSC  -  Failure  Definition  Scoring  Criteria 

■  Failure  Categories 

■  Inherent  vs.  Induced  Reliability 

■  Mission  Profile  and  Life  Variable 

■  Data  Grouping  and  Modeling 

■  Instantaneous  vs.  Cumulative  Reliability 


•  MGS  Lesson  Learned  -  DFR 
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MGS  -  Systems  Engineering  Approach 


•  Integrates  All  Reliability  Tasks 

•  Redirects  Tasks  Toward  a  Single  Objective 

•  Crosses  Boundaries  Affecting  Operational  Reliability 

•  Provides  Program  Manager  Authority,  Funding,  and 
Focus  on  Engineering,  Processes,  Documentation, 
Training,  Manufacturing,  and  Testing  for  Reliability 

•  Approach  Provides  Metrics  that  can  be  Measured 
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SE  Approach  to  Reliability 


Increase  Design  Effectiveness  Using 
Robust  Design  Methodology 


r  Potential  MTBF 


Modeling 
Allocation 
Prediction 
FMEA 
Parts  Program 
FRACAS 
Failure 
Prevention  & 
Review  Board 
Verification 


Design  Phase 


<4 


Manage  Growth  Potential 


Higher  Initial  MTBF 
At  Start  Of  Test 

Failure  Prevention 
Failure  Categorization 
Timely  Corrective  Actions 


RG/DT 
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Design  for  Reliability  Management 
Focuses  on  Failure  Prevention 


Requirements 

Review 


Performance 

Requirements 


Environmental* 

Requirements 


Reliability 

Requirements 

Definitions 


Safety 

Requirements 


Maintainability 

Requirements 


Support 

Requirements 


Analyses 


Testing 


Verification 


Validation 


Reliability  Growth 


IRGT,  FRACAS 


Design  for 
Reliability 

Identify 
Risk  Modes 


Update  Status 


Reliability 

a 

Growth 

A 

In  Design 

A 
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Stryker  -  Mobile  Gun  System 

Failure  Prevention  and  Resolution  Implementation 
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Agenda 


•  What  is  MGS 

•  Success  Factors  of  MGS  PVT 


Program  Management  -  Integrated  Team 

System  Engineering  and  Reliability  Attainment 

Reliability  Data  Analysis  -  RGA 

■  FDSC  -  Failure  Definition  Scoring  Criteria 

■  Failure  Categories 

■  Inherent  vs.  Induced  Reliability 

■  Mission  Profile  and  Life  Variable 

■  Data  Grouping  and  Modeling 

■  Instantaneous  vs.  Cumulative  Reliability 

•  MGS  Lesson  Learned  -  DFR 
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Reliability  Data  Analysis 


•  Proper  Reliability  Assessment  is  a  key  for  the 
program  success  at  PVT 

•  Reliability  Assessment  must  be  discussed  up 
front  and  consensus  should  be  reached  on: 

71  FDSC  -  Failure  Definition  Scoring  Criteria 
71  Failure  Categories 

■  Inherent  vs.  Induced  Reliability 
71  Mission  Profile  and  Life  Variable 
71  Data  Grouping  and  Modeling 
7i  Instantaneous  vs.  Cumulative  Reliability 


16 


GENERAL  DYNAMIC 

Land  Systems 


Approved  for  Public  Release,  Distribution  Unlimited,  GDLS  approved,  log  2008-04,  dated  02/11/08. 


FDSC  -  Failure  Definition  Scoring 
Criteria 

•  FDSC  is  Contractual  Document  that  defines 
71  Failure/non-Failure  Event 

71  Test  related  Event 

71  Severity  of  Failure  as  it  relates  to  the  Mission 
71  Cause  of  the  Failure 

•  FDSC  is  prepared  as  required  by  Army 
Regulation  70-1,  Army  Acquisition  Policy. 

•  FDSC  is  being  used  through  out  the  test  for 
Scoring  purposes,  hence  it  is  a  major  document 
for  Reliability  Assessment 
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Failure  Categories 


•  Performance  FM  -  FM  is  repeatable  with  100%  probability  of 
failure  for  the  given  procedure/conditions.  (Example:  TDS 
overheating) 

•  Software  FM  -  same  as  above,  but  software  related. 

•  Quality  FM  -  happens  when  vehicle  is  not 
built/maintained/operated  as  designed  and  is  not  repeatable  after 
fixing  (probability  of  failure  =0%).  Can  be  broken  down  into  Initial 
Quality,  Maintenance,  Operator  error,  etc.  (Example:  Improperly 
installed  harness,  turret  lock  bended,  etc.) 

•  Potential  Reliability  FM  -  happens  when  vehicle  was 
built/maintained/operated  as  designed/intended;  probability  of 
failure  is  greater  than  0%  and  less  than  100%;  usually  happens 
due  to  wear  out,  environment,  insufficient  design,  manufacturing 
variability,  etc. 
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Failure  Mode  Categorization  Process 
Inherent  vs.  Induced  Failure 


Categorize  Failures  and  take 
Relevant  Management  Actions 
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Data  Grouping 


Known  Equivalent  Time  Unknown  Equivalent  Time 
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Rounds  and  Miles  Accumulation  per 
Vehicle  vs.  Calendar  Time 


Cum  IVfles  vs  Calendar  Time 


KET  Model  can  be  useful  in  the 
beginning  of  the  test  when  vehicles 
have  not  accumulated  enough 
mileage  and  rounds. 


UET  model  takes  into  account 
any  discrepancies  between 
different  vehicles  following 
through  the  test  in  calendar  time 


Calendar  Time 
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Cum.  Number  of  Failures 


Crow/AMS  AA  Model 


ReliaSoft's  RGA  6  PRO  -  RGA.ReliaSoft.com 

Cumulative  Number  of  Failures  vs  Rounds 


Rounds 


Cum  Number  of  Failures 

E(N)  =  X-TP 

Cum  Failure  Rate 

rc  = 

■EW=x.t» 

T 

Cum  MTBF 

MTBFc={rcy  ={X-T^y 

Inst  Failure  Rate 
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Cumulative  vs.  Instantaneous 

Reliability 

•  Reliability  growth  on  the  Development  test  is 
the  result  of  Corrective  Actions. 

•  Estimating  Reliability  of  the  product  by  taking 
the  Cumulative  reliability  (total  number  of 
failures  /  total  time  on  the  test)  does  not  take 
into  account  the  growth  on  the  test. 
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Idealized  Growth  Curve  and  Observed 
Parametric  Curve  for  Demonstrated  Instantaneous 
MRBSA 
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Agenda 


•  What  is  MGS 

•  Success  Factors  of  MGS  PVT 


7i  Program  Management-  Integrated  Team 
7i  System  Engineering  and  Reliability  Attainment 
71  Reliability  Data  Analysis  -  RGA 

■  FDSC  -  Failure  Definition  Scoring  Criteria 

■  Failure  Categories 

■  Inherent  vs.  Induced  Reliability 

■  Mission  Profile  and  Life  Variable 

■  Data  Grouping  and  Modeling 

■  Instantaneous  vs.  Cumulative  Reliability 

GS  Lesson  Learned  -  DFR 
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DFR  Process  Elements 


•  Boundary  Diagram  /  System  Block 
Diagram 

•  Interface  matrix 

•  P-Diagram 
• DFMEA 

•  Reliability  &  Robustness  Metrics 

• DVP&R 

•  Reliability  Demonstration  Metrics 
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DFSS  (DCOV)  Flow  of  Analysis  &  Tools 
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Design  For  Reliability  Map 


GENERAL  DYNAMICS 

Land  Systems 


Control 
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R 


MIL-HDBK-1 89  RGA  Method 
MGS  MEP  PVT  Instantaneous  MRBSA 


Demonstrated  Instantaneous  MRBSA  for 
decision-makers 


Crow  (NHPP) 


Failure  Rate  continues  to  decrease,  thus 
demonstrating  substantial  reliability  growth 
in  PVT 


*  Growth  Rate  is  0.4 

•  RGA  Methodology  was  developed  and 
agreed  by  RAM-T  Community 


\ 


Continuing  the  effort  to  ensure  MGS  reliability  growth 

Systems  Engineering  Process  continues  to  be  worked  “24/7” 

GDLS  Senior  Leadership  briefed  on  a  daily  basis 

Focus  on  implementation  of  Corrective  actions  on  both  the  Test 
Vehicles  and  the  Fielded  vehicles 


GDLS  teams  at  our  vendors  to  work  failure  analysis  and  ensure 
MGS  gets  their  top  priority 


Outside  experts  on  reliability  and  quality  regularly  review  our 
processes  in  engineering  and  Manufacturing  so  we  keep  getting 
better 


•  Sustained  decrease  of  MGS  Failure  Rate 
suggests  infant  mortality  region  is  passed 
and  design  is  maturing 


j 


ReliaSoft's  RGA  6  PRO  -  RGA.ReliaSoft.com 


Instantaneous  Failure  Intensity  vs. Rounds 


TO 

to 
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Rounds 


Beta=0.5945 
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Keys  to  Success 


•  Program  Management  forms  Integrated 
Team  (Material  Developers, 
Tester/Evaluators,  User)  that  has  clear 
priority  and  focus  on  Reliability  with  clear 
understanding  of  Evaluation  Criteria  and 
Test  Methods  up  front. 

•  System  Engineering  assembles  Reliability 
tools  into  Disciplined  processes  and  Working 
Organizations 

•  Reliability  Assessment  is  reached  through 
in-depth  analysis  and  consensus  between  all 
involved  parties 


r  ^ 

Program  Management  +  System  Engineering  +  Reliability  =  Success 
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Questions  and  Discussion 
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•  Dmitry  Tananko,  Ph.D. 

71  General  Dynamics  Land  Systems 
7i  Tel.:  (586)  634-5071 
7i  E-mail:  tanankod@qdls.com 


34 


GENERAL  DYNAMIC 

Land  Systems 


Approved  for  Public  Release,  Distribution  Unlimited,  GDLS  approved,  log  2008-04,  dated  02/11/08. 


Project  Manager 


NDIA  11th  Annual 
Systems  Engineering 
Conference 

22  Oct  2008 


Establishing  a  Systems 
Engineering  Center  of 
Excellence  within  PEO  GCS 


Mike  Phillips 

PM  MBE  Systems  Engineer 


TACOM  LCMC 

TACOM  Life  Cycle  Management  Command 


Department 
of  Army 
(DA) 


Assistant  Secretary 
of  the  Army 

for  Acquisition,  Logistics, 


PEO  PEO  PEO  Acq  ILSC  Depots  ARDEC 

Soldier  CS&CSS  GCS  Center  and 

Arsenals 


Army 

Materiel 

Command 

(AMC) 


I 


TARDEC  NSRDEC 


The  TACOM  LCMC  unites  all  of  the  organizations  that  focus  on  Soldier  and 
Ground  Systems -  The  PEOs  and  PMs  are  able  to  work  as  an  integral  part  of 
the  Logistics  and  Technology  efforts  of  the  LCMC,  while  enterprise  level 
partnerships  are  maintained  with  the  Research,  Development,  and 
Engineering  Centers  (RDECs). 


TACOM  LCMC  Play  book 


ri 


TACOM  LCMC  Vision 


Providing  our  warfighters  with  pra 

overwhelming  lethality,  survivability, 
mobility,  and  sustainment  for 
battlefield  dominance,  now  and  in  the  tffl 
future.  ’•*3 


TACOM  LCMC  Mission 

Develop,  acquire,  field,  and  sustain 
soldier  and  ground  systems  for  the  '*mm 

warfighter  through  the  integration  of 
effective  and  timely  acquisition, 
logistics,  and  cutting-edge  technology.  "  * 
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PEO  GCS 

Program  Executive  Office  Ground  Combat  Systems 


Heavy  Brigade  Stryker  Brigade  Joint  Robotics  Systems  Mine  Resistant  Joint  LWH  155mm  Modular  Brigade 
Combat  Team  Combat  Team  (Army  &  Marine)  Ambush  Protection  (Army  &  Marine)  Enhancements 


COL  P  Lepine  COL  R.  Schumitz  Col  J.  Braden  (USMC)  COL  K.  Peterson 


Mr.  J.  Shields 


COL  J.  Wendel 


Vision 


Exceed  Warfighter  expectations  as  the  Army's  Lifecycle  Manager  and  systems  integrator  for  current  and 
future  Ground  Combat  Systems. 


Mission 


Manage  the  development,  systems  integration,  acquisition,  testing,  fielding,  sustainment  and 
Improvement  of  ground  combat  systems  in  accordance  with  the  Army's  initiatives  to  provide  mission- 
capable  systems  to  the  Warfighter  while  meeting  cost,  schedule  and  Performance  goals. 
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Supporting  the  Army  Vision  Requires 
Synchronized  Modernization  WHY? 


•  GCS  Platform 
infrastructure  has 
remained  relatively 
constant  since  the  last 
development/improvement 
program 

•  Requirements  are  evolving 
and  expanding  which 
requires  integration  of  new 
capabilities 

-  New/Updated 
CDDs/CPDs  under 
development 

-  Integrating  new 
capability  to  already 
strained  power,  space, 
and  weight  claims 

•  Integrating  more  in  current 
vehicle  configuration 
impacts  crew  and  vehicle 
capability 


on 


m 


We  are  at  the  degradation  point 
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Systems  Engineering  Center  of  Excellence 
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SECOE  Description 


Systems  Engineering  Center  Of 
Excellence  is  an  operational 
organization  infused  with  common 
SE  processes  and  tools  to 
optimize  execution  of  acquisition 
programs 

DEVELOPMENT  TENETS: 

•  Comprehensive  system-of-systems 
integration  methodologies 

•  Support  senior  management  fact-based 
decision  making 

•  End-to-end  processes  that  are 
tailorable,  scalable,  &  portable 

•  Focus  on  PEO-wide  problem  sets 

•  Maximize  common  tools  and  processes 


Systems  Engineering 


A  branch  of  engineering  whose 
responsibility  is  creating  and 
executing  an  interdisciplinary 
process  to  ensure  that  customer 
and  stakeholder's  needs  are 
satisfied  in  a  high  quality, 
trustworthy,  cost  efficient  and 
schedule  compliant  manner 
throughout  a  system's  entire  life 
cycle,  from  development  to 
operation  to  disposal. 

-  International  Council  on  Systems 
Engineering  (INCOSE) 


7 


SE  Capability  Growth 


Implementation  Elements 


SECOE  COMPONENTS 


(  SECOE  ] 

< 


Processes 

A  set  of  formalized 
methodologies  that  guide 
program  execution 


Tools 

Software  applications 
that  enable  the  execution 
of  processes 


Training  ' 

Increasing  knowledge  of 
SE  processes  and  SE 
ability  of  the  staff 
executing  the  acquisition 
programs 

J  V _ J 


\ 


Standard  Operational  Procedures 

Procedures  that  describe  how  processes,  tools,  and  training  are  applied 

to  bring  about  an  SE  capability 


Transition  Plan 

Plan  that  moves  PEO  from  its  initial  state  to  desired  SE  culture 


Resources 

The  personnel,  funding,  and  facilities  necessary  to  execute  the  processes,  tools,  and  training 
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Near  Term  Plans  Current  Efforts 


Growing  Core  Capabilities 


r 


PROCESSES 


TOOLS 


TRAINING 


A 


•  IMP/IMS  Development 

•  IMS  Maintenance 

•  Capability  Alignment 

•  SEP  Development 

•  Risk  Management 


•  Tools  Plan 

•  Risk  Management  Tool 

•  Requirements  Mgmt 

•  Integrated  Scheduling 


•  Training  Plan 

•  SE  Curriculum 


STANDARD  OPERATIONAL  PROCEDURES 


•  Systems  Engineering  Integration  Team  Review  and  Approval  SOP 

•  Risk  Management  Process/Tool  Application  SOP 
•PEO  IMP/IMS  SOP 


< 


V 


Technical  Reviews 
Requirements  Mgmt 
Unit  Set  Fielding 
Tech  Readiness  Assmnt 
Mfg  Readiness  Assmnt 


Fielding  Management 
Automated  IMP 
Template 

Configuration  Mgmt 
Data  Management 
Modeling  and  Simulation 
Architecture  Tools 
Reliability  Tools 


•  Workforce  SE 
Orientation 

•  Pilot  Training  Program 

•  Professional  Affiliations 

•  SE  Training  Coordinator 

•  Academic  Partnerships 

•  SE  Library 


Integrated  Scheduling 

Aligning  Across  Platforms 


Individual  Schedules 

Scheduling  Tools 

PEO  GCS 

Differing  formats 
Differing  detail 
Differing  software 


5 

* 

-- »  .....  •  • 
^  - i - — »  ■  - - !  -  •-  •  -  -  -  » ■ 

- — g 


mm 


Built  using  off-the- 
shelf  software  SOPs 
being  developed 


SefrecMe 
Maintenance  Toot 


Enterprise  Project 
Management 
(EPM) 


MS 

Project 


PEO  GCS 
Knowledge  Center 


Integrated  Master  Schedule 


Scheduling  Opportunities 


PEO  and  PMs  gaining  better  insight  across 
programs 

Focusing  on  sustainment  &  modernization 
Managing  Schedule  Risk 
Identifying  Commonality  Opportunities 
Supporting  “What  If  Drills” 

Synchronizing/Standardizing  schedules  across  PEO 
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Risk  Management 


PEO  GCS  Risk  Management  Process 


Task  1.0-1 

%sk  Planning 

Task  2.0  -  Risk  Assessment 

Task  3.0  -  Risk  Handing  Plan 

Task  4.0  -  Continuous  Risk  Monitoring  | 

^4 

1.2 

1 

:  No/CXv. 

Resource 

| - V  «**“*-«.  I 

ifsbu-ihi 

u _ S’ 

Improving  Risk  Management 


Proactively  Managing  Risk 
PEO  and  PMs  using  a 
common  understanding  of 
program  risk 

Supporting  “What  If  Drills” 


PEO  GCS  risk  management  tool  is 
being  used  to  automate  the  risk 
management  process 

Integrated  in  the  PEO  GCS  Knowledge 
Center 

The  process  is  based  on  and  aligns  with 
DOD  risk  guidance 

The  tool  is  portable  and  tailorable  to 
other  PEOs 


PEO  GCS  Risk  Management  Tool 


Risk  Info  Sheet  Risk  Info  Team  Detailed  Risk  Analysis  Handling  Plan(s)  Related  Projects 


Risk  Assessment 


Likelihood  3 1 


2  3  4 

Consequence 


GEE]  I C3naei  1 

Risk  ID: 

Risk  Title: 

RISK-2008-Engineering  -  Widget  Form  Factor  Exceeds  Available  Spaceclaim 

* 

Status 

Baselined  v 

Open  Date: 

8/17/2008  |  * 

Last  Reviewed  On  Date: 

10/3/2008 

WBS  #: 

N/A 

IMP,: 

CDR-003-002 

Risk  Lead: 

|  Phillips.  Mike  H  * 

|*  required  field 
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SE  Analyses  Processes  &  Tools 


What  does  it  do? 


Which 

subsystem  does 
it  impact? 


Functional 

•  CONOPS 

•  OV-1 

•  Use  Case  Diagrams 

•  Use  Case  Text 


Division  of 
Responsibilities 

•  Sequence  Diagrams 
• FFBDs 

•  Spreadsheets 


Human  Factors 

•  Operator  Interface 

•  Roles 

•  PDDs 


How  does  a  user 
perform  the 
behavior? 


Modernization 

& 

Commonality 
Analysis 


On  what  assets 
is  the  behavior 
performed? 


Cross  Platform  analysis 

•  Physical  Block  Diagrams 

•  Align  Schedules 


Performance 

•  Timing 

•  TPMs 

•  Performance  Analysis 


How  well,  how  fast  and 
at  what  frequency? 


Life  Cycle  Analysis 

•  Life  Cycle  Costs 

•  Program/System  Risks 


What  is  the  total  cost 
impact  to  the 
program? 


Systems  Engineering-Based  Analysis  Processes  Being  Improved 


Two- Level  Platform  Analysis 


System  Wide  Analysis  of  Potential  Components 


Move 

Shoot 

Comm 

Survive 


Component 

Performance/ 
Capability 

Needed 
Power 

Space  Claim 
Thermal 


Vehicle 


Performance 

Needs 

Available 
Power 
Space  Claim 
Thermal  Limits 


Needed  Vehicle 
Infrastructure 
Improvements 


Component 
Mods  to  support 
vehicle  needs 


Coordinated 

Vehicle 

Modernization 

Plans 

& 

Component 

Development 

Plans 


Commonality  Analysis 


Commonality 

Optimization 


SE  Training 


SECOE  Training  Objectives: 

-  Train  a  SE  qualified  workforce 

•  Trained  to  understand  systems  engineering 

•  Trained  to  manage  systems  engineering 

-  Increase  visibility  into  available  SE  training  and 
certifications 

-  Establish  single  training  tracking  tool  for  SE  training  & 
certifications 

•  Working  with  DAU  to  customize  &  implement  PEO  GCS  training 

-  Available  to  PEO  CS/CSS,  TARDEC,  and  TACOM 


World-Class 

SE  TRAINING 

SE  PEOSE 

Support  Expertise 


Time  — ► 

Evolving  the 
Workforce  Over  Time 


-  Focusing  on  growing  number  of  Level  III  certified  SPRDE,  Program 
Systems  Engineers 

•  Working  with  professional  organizations,  academia 

-  Aligning  and  educating  workforce  on  available  SE  certifications  and 
degree  programs  for  those  interested 

-  Utilize  existing  TACOM  training  databases  (e.g.,  TEDS)  to 
implement 

Near  Term  Timeline: 

-  Sep  08:  Draft  Training  Plan 

-  Sep  08:  Draft  Training  Curriculum 

-  Nov  08:  SE  Workforce  Briefing  Complete 

-  Nov  08:  Pilot  Training  Delivery 

-  Dec  08:  SE  Library  Initiated 

-  Jan  09:  Professional  Development  Opportunities  Identified 

-  Feb  09:  SE  Training  Process  Approved 


PEO  GCS 

System  Engineering  Curriculum 

2008 
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Approval  Process 


Process  Flow 


Process  Steps 


Approval  to 
Implement 


Phase  1 : 
Need  and 
Concept 


1.1  Identify  SE 
Product  Need 


1 .2  Define  Scope 
and  High-level 
Solution  Concept 


1.3  Present  Draft 
SE  Project 
Directive  to  SEIT 
for  Approval 


A 


Phase  2: 
Draft 

Development 


2.1  Form  IPT  to 
Develop  Product 


2.2  Develop  Draft 
Product 


2.3  Present  Draft 
Product  to  SEIT  for 
Guidance 


Phase  3: 
Final 

Development 


3.1  Develop  Final 
Product 


3.2  Develop 

Associated 

Training 


Phase  4: 
Implementation 


4.1  Add  Product  to 
the  PEO  GCS 
Baseline 


4.2  Deliver  Training 
to  the  User 


3.3  Present  Final 
Product  to  SEIT  for 
Approval 


Systems  Engineering  &  Integration  (SEIT)  Membership:  PEO  Lead  SE  (chair),  PM  Lead  SEs,  CIO 
Systems  Engineering  Advisory  Council  (SEAC)  Membership:  PEO  Lead  SE  (chair),  PMs,  CIO 
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SECOE  Steady  State 


SEs  and  Other  Users 


PEOlPTsand  SECOE 


Process/Tool 

Improvement 


SEIT/SEAC/PEO  GCS 


Review  /  Approval 


Feedback  Loop 


SEs  and  Other  Users 


A  Lifecycle  of  Continuous  Improvement 


PEO 


PEO/PM  Benefits 


PM  Benefits 


TACOM  Community 


SECOE  Stakeholder  Benefits 


PEOs  executing  acquisition 
programs  with  greater 
efficiency  while  reducing 
turbulence  and  disruption  to 
the  Unit 


Provides  synchronized  views 
across  the  PMs 


Benefits  Growing  systems 
engineering  capabilities  within 
the  community  and  building 
for  the  future 


Suite  of  processes,  tools,  and 
training  to  enable  more 
efficient  program  planning  and 
execution  in  terms  of  cost, 
schedule,  and  performance 


Army  Benefits 


APEO  SEIO 


PMs 


Other  Orgs 
(PEO  CS/CSS, 
TARDEC,...) 
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The  Future 


•  Update  on  PEO  GCS  progress  will  be  provided  at 
NDIA  12th  Annual  Systems  Engineering  Conference 

•  In  the  meantime,  contact  me  if  you  want  to: 

-  Contribute  good  ideas  to  our  effort 

-  Steal  good  ideas  from  our  effort 


Mike  Phillips 

mike.h.phillips@us.army.mil 

586.574.8879 
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Two-Step  Methodology  to  Reduce  Software  System 

Requirement  Defects 


Presented  to 

NDIA  Systems  Engineering  Conference 

21  October  2008 


Presented  by 

Robert  J.  Kosman 

Operational  Systems  Division/1552 

(401)  832-8571,  robert.kosman@navy.mil 

APPROVAL  FOR  PUBLIC  RELEASE;  DISTRIBUTION  IS  UNLIMITED 


Software  System  Development 


NEWPORT 


“Typical” 

Software  System  Development 


□  Waterfall  /  Incremental  model 

□  Spiral  model  similar  for  a  spiral 

□  Implies  a  sequential  process  to 
resolve  problems  (defects) 

□  Does  not  provide  an  adequate 
illustration  of  defect  impacts 
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AAA 


NEWPORT 


Software  System  Development 


“Realistic” 

Software  System  Development 


□  Added  links  backwards  to 
reflect  origin  of  defects 


□  Omitted  links  other  than  those 
back  to  the  first  phase  - 
software  system  requirements 
development 

□  Rework  caused  by  defects 
can  impact  cost  and  schedule 
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WARFARE  CENTERS 


Software  System  Development 


NEWPORT 


DEFECTS  AND  REWORK 

$  Rework  caused  by  defects  can  impact  cost 
and  schedule 

$  The  later  a  defect  is  found,  the  greater  the 
cost  to  correct 

$  Defects  found  and  fixed  in  later  phases  of 
development  can  cost  up  to  lOOx  the  cost  to 
correct  if  detected  in  early  phases 

■  Software  Specifications 

■  S/W  designs,  code,  test,  documentation 

■  Integration,  T&E  plans  and  procedures 

■  Integrated  Logistics  Support  (ILS)  products 
(Operator  /  User  manuals,  Training  materials, 
etc) 

■  Distribution  costs 

■  Change  documentation 


REQUIREMENT  DEFECTS 

■  Impacts  all  phases  and  products 
(“Negative  Ripple  Effect”) 

■  Most  costly  to  correct 

■  Cause  delays  in  schedule  and 
product  delivery 

■  Initial  system  may  have  reduced 
capability  and  functionality,  and 
most  likely  operational  limitations 

■  Usually  require  formal 
documentation  to  correct,  e.g., 
Engineering  Change  Proposal 
(ECP) 


DEFECT  CORRECTION  EXPENDS  RESOURCES  AND  FUNDS 
REQUIRED  FOR  PLANNED  SYSTEM  CAPABILITIES 
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S/W  System  Requirement  Defects 


PROPOSED  METHOD  TO  REDUCE  SOFTWARE  SYSTEM  REQUIREMENT  DEFECTS 


□  When: 


•  Focus  on  software  development  phase  of  acquisition;  initial 
development  or  maintenance  phase 

•  Prior  to  Software  Specification  Review  (SRR)  and  Preliminary  Design 
Review  (PDR) 

»  Low-level,  defect  detection  process  prior  to  high-level,  program  milestone 
review 

»  Process  generates  better  products  input  to  SRR  and  PDR,  or  an 

Engineering  Change  Proposal  (ECP)  during  life-cycle  maintenance  phase 

•  Used  during  system  software  specification  generation,  i.e.,  during 
translation  of  high  level  Performance  Specification  and  user 
requirements  (CONOPS)  or  User  Requirements  Document  into  low- 
level  Software  Requirement  Specifications  (SRSs) 

•  Systems  Engineering  (SE)  organizes  and  runs  the  defect  detection 
process 

»  SE  oversees  technical  aspects  of  the  entire  system  acquisition,  including 
processes  to  find  defects  in  ALL  products 
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S/W  System  Requirement  Defects 

NEWPORT 


PROPOSED  METHOD  TO  REDUCE  SOFTWARE  SYSTEM  REQUIREMENT  DEFECTS 


□  How: 

•  Analysis  on  past  defects  identifies  two  basic  types  of  s/w  system  requirement 
defects 

•  The  defect  that  is  unintentionally  introduced  into  the  s/w  system  requirement 
specifications  during  specification  generation 

»  Ambiguous  text 

»  Equation  errors  (algorithms) 

»  Figure  errors  (functional  and  processing  flows) 

»  Table  errors  (wrong  units,  input  ranges,  etc.) 

»  Connectivity  and  inconsistency  issues 

»  Missing  or  incomplete  requirements 

•  The  defect  that  causes  effort  to  be  expended  producing  unnecessary,  incorrect 
or  unwanted  functionality 

»  “Bells  and  whistles” 

»  Inadequate  graphical  user  interface  (GUI) 

-  Systems  are  becoming  more  user  interface  driven  (COTS)  so  the  proposed  GUI  should  be 
included  in  the  s/w  specification 

Need  to  eliminate  user  comments  like,  “system  should  work  this  way” 


CAUTION 


S/W  engineers  will  fill  in 
the  ‘holes’  and  ‘gaps’ 
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S/W  System  Requirement  Defects 


NEWPORT 


□  How 


Develop  methodology/process  to  address  both  types  of  s/w  system 
requirement  defects 

First,  tackle  the  mistakes  made  translating  P-Spec  and  User 
specifications/CONOPS  into  functional  flows  and  the  GUI 

»  “Bells  and  whistles” 

»  Unnecessary,  incorrect  or  unwanted  functionality 

Second,  tackle  the  mistakes  made  generating  the  s/w  system 
requirements  specifications 

»  Usual  mistakes  made  producing  specifications,  e.g.,  ambiguous  text,  etc. 
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S/W  System  Requirement  Defects 


PROPOSED  METHOD  TO  REDUCE  SOFTWARE  SYSTEM  REQUIREMENT  DEFECTS 


□  Introduce  a  two-step  methodology  for  s/w  system  requirements 
clean-up 

1:  Operational  Demonstration  (OP-DEMO)  of  the  User  Requirements 
»  Visual  demonstration  of  proposed  GUI  and  functional  flows 
»  Allows  evaluation  of  system  functionality  prior  to  development 

2:  S/W  Inspection  conducted  on  software  requirement  specifications 

»  Rigorous  review  originally  developed  for  s/w  but  can  be  applied  to  any 
“readable”  products 
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NEWPORT 


Step  1 :  OP-DEMO 


PROPOSED  METHOD  TO  REDUCE  SOFTWARE  SYSTEM  REQUIREMENT  DEFECTS 


□ 


□ 


□ 


Visualization  of  the  User  Requirements 

•  Operability  and  functional  flow 

•  Graphical  User  Interface  (GUI) 

•  Target  Machine  or  other 

Different  levels  of  OP-DEMO 


•  Operability  features  and  functional  flow 

•  Operability  features  and  functional  flow 
with  limited  processing  (e.g.,  algorithms) 

Form  of  Software  Rapid  Prototyping 

•  Disposable  code 

•  Developed  FAST  using  appropriate  tools 

•  User  involvement  early  -  during  s/w  requirements 
phase 
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Step  1 :  OP-DEMO 

NEWPORT 


PROPOSED  METHOD  TO  REDUCE  SOFTWARE  SYSTEM  REQUIREMENT  DEFECTS 


□  Wrong  Concept  of  OP-DEMO  (prototyping) 

•  Target  machine  is  always  utilized 

•  Deliverable  code 

•  Considered  ‘full’  system  operability 

•  User  involvement  in  later  phases 

•  Fix  problems  in  maintenance  phase 


CAUTION 


OP-DEMO  is  Similar  to  Prototyping  and  Prototyping 
Means  Different  Things  to  Different  People 
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OP-DEMO  Benefits 


NEWPORT 


□  Involves  the  User  during  the  early  phases,  as  opposed  to  the 
later  phases  or  after  system  delivery 

□  Eliminates  unnecessary  and  incorrect  functionality  and  helps 
prioritize  remaining  functionality 

□  Provides  a  working  model  of  intended  operation  for  reference, 
as  well  as  tool  to  allow  parallel  development  of 
operator/training  materials 

□  Identifies  areas  of  uncertainty  for  risk  management 

□  Promotes  faster  and  more  accurate  s/w  system  specification 
writing 
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Step  2:  Requirement  Inspection  (Rl) 

NEWPORT 


PROPOSED  METHOD  TO  REDUCE  SOFTWARE  SYSTEM  REQUIREMENT  DEFECTS 


□  “Software  Inspection”  applied  to  the  Software  System  Specifications 

□  Not  like  an  informal  ‘Code  Walkthrough’ 

□  Formal,  intensive  review  process  designed  to  detect  errors 

•  Ambiguous  text 

•  Equation  errors  (algorithms) 

•  Figure  errors  (functional  and  processing  flows) 

•  Table  errors  (wrong  units,  input  ranges,  etc.) 

•  Connectivity  and  inconsistency  issues 

•  Missing  or  incomplete  requirements 

□  Basic  characteristics 

•  Team  approach,  with  assigned  roles  (reader,  moderator,  author) 

•  Standards  of  conduct 

•  Collect  metric  data 

•  Criteria  for  Quality 

Documented  results  indicate  up  to  85%  of  design  and  code  errors 
can  be  detected  by  “Software  Inspections” 
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Step  2:  Requirement  Inspection  (Rl) 

NEWPORT 


PROPOSED  METHOD  TO  REDUCE  SOFTWARE  SYSTEM  REQUIREMENT  DEFECTS 


□  Team  Members 

•  Software  Engineer  (Lead) 

•  System  Engineer 

•  User  (or  ILS  person) 

•  Test  Engineer 

□  Multiple  teams  (2  or  3)  detect  more  defects  (N-Fold  Inspection) 

•  Small  %  of  duplicate  defects  found  between  multiple  teams 


Multiple  discipline  involvement  ensures  consistent  interpretation 
of  software  system  requirements  across  phases 
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Requirement  Inspection  Benefits 


NEWPORT 


PROPOSED  METHOD  TO  REDUCE  SOFTWARE  SYSTEM  REQUIREMENT  DEFECTS 


□  Ensures  User  requirements  are  accurately  specified 

□  Ensures  developer  requirements  are  accurately  specified 

□  Real-time  metric  data  collection  identifies  areas  of 
improvement  w/  specification  generation 

□  Errors  corrected  in  single  pass  versus  iterative  correction 
process 

□  Detects  errors  associated  with  all  phases  of  the  Development 

□  Low  cost  /  defect  ratio 

□  Reduces  software  development  costs  by  detecting  errors  early, 
avoids  REWORK 
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Requirement  Inspection  Benefits 


WARFARE  CENTERS 


NEWPORT 


PROPOSED  METHOD  TO  REDUCE  SOFTWARE  SYSTEM  REQUIREMENT  DEFECTS 


A 


SCHEDULE 


Impact  of  Rl  on  Development  (modified  from  [1]) 

[1]  Fagan,  M.E.,  “Advances  in  Software  Inspections,”  IEEE  Transactions  on  Software  Engineering ,  Vol  SE-12,  No.  7,  July  1986 
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NEWPORT 


Case  Study 


PROPOSED  METHOD  TO  REDUCE  SOFTWARE  SYSTEM  REQUIREMENT  DEFECTS 


□  Two  extensive  upgrades  to  an  existing  system  -  approx  100 
KSLOC  each 

•  Existing  system  was  really  a  “prototype/experimental”  system  delivered 
as  a  production  system;  so  had  to  fix  in  Maintenance  phase  via  ECPs 

•  First  upgrade  did  not  use  2-Step  Methodology  to  reduce  Software 
System  Requirement  Defects;  second  upgrade  did 

•  Software  System  Specifications  for  first  upgrade  were  developed  by  SE 
with  only  informal  reviews,  and  significant  portion  of  user  interface  was 
“TBD/TBS” 

•  Software  development  team  was  already  using  Software  Inspection 
during  development  so  extensive  defect  metric  data  was  collected  during 
both  upgrades 

•  Causal  analysis  was  conducted  on  all  defects  found  to  determine  origin 
of  defect 

•  Both  types  of  OP-DEMO  were  utilized  on  second  upgrade  (algorithms); 
2-Fold  Rl  also  used  on  second  upgrade 
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BESS  Case  Study 


WARFARE  CENTERS 


NEWPORT 


PROPOSED  METHOD  TO  REDUCE  SOFTWARE  SYSTEM  REQUIREMENT  DEFECTS 


Upgrade  1  Observations 


Requirement  Defects  By  Phase  -  UPGRADE  1 


REQ  DESIGN  CODE  TEST  PTRs 


PHASE 


$  Informal  reviews  found  some 
defects  but  not  enough 

$  Defects  found  during  Design  and 
Code  could  have  been  found  by 
Rl 

$  Defects  found  during  computer- 
based  Test  and  Post-delivery 
could  have  been  found  by  OP- 
DEMO 

$  Rework  caused  schedule  delays 
and  end  product  had  reduced 
functionality 

$  Defects  required  multiple 
updates  to  s/w  system  spec 
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BESS  Case  Study 


WARFARE  CENTERS 


NEWPORT 


PROPOSED  METHOD  TO  REDUCE  SOFTWARE  SYSTEM  REQUIREMENT  DEFECTS 


Upgrade  2  Observations 


Requirement  Defects  By  Phase  -  BOTH 


REQ  DESIGN  CODE  TEST  PTRs 


$  OP-DEMO  significantly  reduced 
defects  in  computer-based  Test 
and  post-delivery  phases 

$  Rl  significantly  reduced  defects 
in  Design  and  Code  phases 

$  S/W  Requirement  Spec  had  a 
“positive  ripple  effect”  on 
development 

$  Significantly  less  rework  for  2nd 
upgrade  and  product  was 
delivered  on  schedule  w/  full 
functionality 

$  Req  defects  were  less  severe 
and  were  easily  fixed 


PHASE 


Robert  Kosman.  Naval  Undersea  Warfare  Center  Division  Newoort.  401-832-8571.  robert.kosman(5)navv.mil 
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Summary 


NEWPORT 


□  Software  system  requirement  defects  can  impact  cost, 
schedule,  and  delivered  functionality  due  to  REWORK 

□  OP-DEMOS  are  useful  in  reducing  defects  that  would  be 
identified  during  computer-based  Test  and  Deployment  phases 

□  Requirement  Inspections  are  useful  in  reducing  defects  that 
would  be  identified  during  Design  &  Code  phases 

□  Improved  s/w  requirement  specifications  can  cut  costs  in  ALL 
s/w  system  development  phases,  including  life-cycle 
maintenance 

□  Combining  OP-DEMO  and  Requirement  Inspection  is  a  low-tech 
approach  to  reducing  s/w  requirement  defects;  is  simple  to 
apply  and  requires  minimal  training 
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NDIA  11th  Annual  Systems  Engineering 

Conference 


“ Daily  Challenges  of  Requirements 

Engineering ” 

October  22,  2008 


Frank  Salvatore 

High  Performance  Technologies,  inc. 

3159  Schrader  Road 

Dover  NJ,  07801 

(973)  442-6436  ext  249 

fsalvatore@hpti.com 


Outline 


□  Requirements  Elicitation 

□  Requirements  Capture  and  Management 

□  Requirements  Traceability 

□  Requirements  Control 

□  Reaching  Consensus 

□  Eliciting  Verifications 

□  Communicating  Requirements 

□  Metrics 


Requirements  Elicitation 


How  do  you  gather  the  requirements? 

□  Interviews 

□  QFD  Workshops 

□  Web  Based  Surveys 

□  Vignettes  and  Scenarios 

□  Questionnaires 

□  Brainstorming  and  Mind  Mapping 

□  Analysis/Derivation 

V  Hazard 

V  Fault  Tree 

V  Sensitivity 

V  Trade  Studies 

□  Existing  Documentation  and  or  Policies 

□  Quality  Assurance  Provisions 

Don't  forget  to  Document  Rational.  It  will  save  you  time 
latter  when  you  will  need  to  defend  the  requirements. 


It  involves  a  lot  of 
research  and  is 
evolutionary! 


Interview  Based  Elicitation 


Using  and  Enterprise  Architecture  approach  one  can  first 
probe  into  Business  Goals  and  Architecture  Principles  buy 
asking  questions  to  understand: 


□  Mission  and  Values  of  your  organization 

□  Understand  importance  (PM  Level) 

□  Understand  organization  structure 

□  Understand  Products 

□  Understand  Customers  and  Stakeholders 

□  Understand  Daily  Activities 


Mostly  used  for  Business  Systems 


Interview  Based  Elicitation 


Project  and  Product  Data  can  be  understood  by 
asking  these  leading  questions 

□  What  are  the  Projects/Products  that  the 
organization  manages? 

□  Who  do  you  interact  with? 

□  What  data  types  do  you  manage? 

□  How  do  you  organize  your  data? 

□  What  data  do  you  view  as  being  most 
important? 


□ 

□ 

□ 


Who  are  the  Customers  for  each  product? 
Who  are  the  stakeholders  for  each  product? 

What  are  the  day  to  day  activities  that  go 
on  for  the  projects  you  choose? 


QFD  Based  Elicitation 


Eng 

ineering  Metrics 
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Requirements  are  Discovered  Thru 
__ _ The  SW  Safety  Process 

I  SOFTWARE  SAFETY  PROCESS  I 


Eliciting  Verification  Methods 


Similar  to  Requirements.  Stakeholders  are 
different.  Methods  are  typically  thru 
Analysis,  Test,  Inspection,  Measurement. 

□  Use  Interview 

□  Use  Questionnaires 

□  Include  Stakeholders  Early  and  Often. 

□  Have  Stakeholders  Peer  Review  Requirements 

□  Use  a  JCCB 


iile  Edit  Insert  Records  Window  Help 


graph: 


ion: 


Interface  Definition 


3. 1.2. 0-1 


TRL: 


r  TRL5 
T  TRL6 
r  TRL7 


I  FT  Name:  Ammo  Handling 

Module  Name  AHR 


jirements:  The  Ammo  Handling  Subsystem  will  interface  with  the  T urret  Structure,  Gun  j|^js  enables  and  disables 

Assembly,  Fire  Control,  Ammo  Suite  and  Secondary  Armament.  AT D /Objective  Force:  reqUjrec|  \\e\^\  Warning- 

Switch  to  View  Mode 


Requirements  Capture  and 

Management 


How  and  where  do  you  store  the  requirements? 

Word  Documents  are  standard.  Tools  are  useful  and  can 
Help.  But  try  to  get  everyone  to  use  them 
consistently!!!!! 

□  Access 

□  Excel 

□  DOORS 

□  RTM 

□  Requisite  Pro 

□  RM  Calibre 

□  etc.... 


Use  Document  Templates  Based  On 
Standards.  Also  IM  is  Important  for  Efficiency. 


Requirements  Management 
Specification  Hierarchy 


Trade  Study 


Verification  C 


Exit  Criteria 


System 

Verification 


Product  C 


System 

Requirements 


/  System 
Level 


Verification  B 


Sub-System  C 


Product  B 


Verification  A 


Sub-System  B 


Product  A 


User 

Documents 


Product 

Level 


Sub-System  A 


Component  B 


Assembly  B 


Component  A 


Assembly  A 


Component 

Level 


Sub-System  Assembly 

Level  Level 


Establish  Hierarchy  and  Naming  Convention ,  Follow  IEEE  Standard 


Document  Outline  is  Standard 
_ Throughout  Project 


Formal  module  712  Sys/Sys  Reqts1  current  4.0  (RFP  12_6_01)  -  DOORS 


File  Edit  View  Insert  Link  Analysis  Table  Tools  User  Help 


b  m 


X  ife  8  V  X  v  m*  m  <  b 


i*  cJ* 


Standard  view 


U 


Level  2 


~3 


\  <na 
□□□ 


il 


Jsername:  fsalvatore 


Exclusive  edit  mode 


*£  S  3  S  S  fg:  ^  n 


ID 

MRAAS  System  Requirements  []  - 

SYSR37  -1  SCOPE 

SYSR38 

2  APPLICABLE  DOCUMENTS 

SYSR39 

>2.1  Government  Documents 

SYSR40 

2.2  Non-Government  Document 

SYSR41 

3  REQUIREMENTS 

SYSR42 

>  3.1  MRAAS  System  Definition 

SYSR48 

>  3.2  Characteristics 

SYSR55 

>  3.3  Design  and  Construction 

SYSR63 

3.4  Documentation 

SYSR64 

>  3.5  Logistics 

SYSR68 

>  3.6  Personnel  and  Training 

SYSR71 

3.7  Major  Component  Characteristics 

SYSR72 

3.8  Precedence 

SYSR73 

4  QUALITY  ASSURANCE 

SYSR78 

5  PREPARATION  FOR  DELIVERY  N/A 

SYSR79 

6  NOTES 

SYSR80 

7  SCHEDULE 

SYSR81 

8  TECHNOLOGIES  TO  INVESTIGATE 

SYSR82 

9  This  section  intentionally  left  blank 

SYSR83 

10  APPENDIX 

A 


Using  Mil-STD- 
490/961 C  standard 
template 

[^Standardized 
Documentation  format 
makes  it  easier  to  find 
what  you  are  looking 
for 


Level  1  User  Requirements 


Level  2  System  Requirements 


Level  3  Product  Requirements 


|  3159  DOORS  Database:  /L3  Prod  -  DOORS 


File  Edit  View  Tools  Help 


E  E  IT  if 


i  m 


a- 3159  DOORS  Database 
[tl  -  ln  Crusader 
-  13  FSAC 
i+  Q  AAN 

a  1*1  Armaments  Server 
+]  |*j  A5SM 

FCMRAAS 
0- -Ji|  FC5 
-i  M  MRAAS 

E  _ |  Analysis 

FF  _ |  Change  Proposal 

0-0  Design 
a-O  Discussions 
+  Q  Links 

+  [ _ |  Management 

5  _ |  Procedures 

£  U3  Reqts 


±1 


m- 

0- 

0- 

0- 

0- 

0- 


LI  User 
L2  Sys 


L3  Prod 


L4  5ubsys 
L5  Asmby 
L6  Comp 


ltl-1  I  T^mnl^hi 


r 


3* 


Name 

1  Type 

|  Description 

_J  Change  Proposal  System  Folder 

System  Information 

J  Discussions 

Folder 

Used  by  Discussion  Forum 

1TA5R 

Formal 

Ammo  Suite  Requirements 

1Ta5V 

Formal 

Ammo  Suite  Verification 

H^MAR 

Formal 

Main  Armaments  Requirements 

HFmAV 

Formal 

Main  Armament  Verification 

Hj^SAR 

Formal 

Secondary  Armaments  Requirements 

ITsav 

Formal 

Secondary  Armament  Verification 

0  Product  Requirements  and 

Verification  Methods. 

IPT’s  Manage  and 

communicate  changes  to 

SEIT. 

J  _<J _ 

_ ■ 

Username:  fsalvatore 


User  type:  Database  Manager 


Level  4-6  Subassembly  to 
Component  Requirements 


Requirements  Traceability 


How  do  you  understand  how  the  requirements 
are  being  satisfied,  are  complete,  are 
accurate,  etc . 

□  Trace  Matrices  are  Typical  and  require  constant  care  and 
feeding  to  maintain. 


□  Use  a  tool  to  manage  your  requirements  and 
capture  traceability  so  you  can  search  and  query 
when  doing  impact  analysis. 


S  More  accurate 
S  More  efficient 
S  More  complete 


If  a  requirement  isn’t  traceable 
to  anything  it  doesn’t  belong!!! 


No  tool  will  automatically 
generate  but  they  will 
preserve  it  once  you  do  it  the 
first  time . 


This  is  Important  when 
performing  Impact  Analysis , 
doing  FCA  and  PCA,  etc .... 


Requirements  Change  Control 


If  a  Requirement  is  changed,  how  do  we  determine 
effects  on  other  Requirements,  Verifications  or 
Schedule  Events? 

□  Use  Inter-IPT  Coordination 

□  Use  Impact  Analysis  &  Visualization  Tools 

□  Use  Formal  Change  Control  Procedures 

□  Attributes 


With  a  tool  you  have  better  and  more  efficient  ways  of 

controlling  the  requirements . 


Follow  a  Change  Proposal 

Process 


Starting  the  Change  Process 


IPT  Member  brings  an  issue  to  attention  of  IPT  Lead 

IPT  Lead  makes  an  initial  determination: 

PURSUE  -  Proposed  change  has  merit  and  is  worth  further 
investigation 

DISCARD  -  Proposed  change  does  not  have  merit  or  is  not 
worth  further  investigation  at  this  time 

If  you  choose  to  PURSUE  the  potential  change: 

1.  Coordinate  with  other  IPT's  to  discuss 

2.  Initiate  working  group(s)  as  needed 


COMMUNICATE  !!! 


Starting  the  Change  Process 


Still  think  a  change  is  needed?  Perform  an 
“Impact  Analysis” 


Step  2 

SitomrtChaige 
Proposal  aid  or 
Suggestions. 
Sibmitadditbial 
CPs  brinpacfcd 
objects. 


col  borate 


Prefect 

Member 


-Problem  Defectd 


Step  3 


Reuiew  CPs  aid 
Siggestbis  br 
Submittal  to  CCB. 


Hold 


Step  4  &  6 


Detimiie  wlblCPsaid 
Siggestbisbreuiewaid 
ssemble  reuiew  package/ 
CP  Let  DEtrbit  actbis. 


Step  5 


CoidictCCB  ReuiewS 
Dspositbi  of  CP  aid 
Siggestbis 


li-Reuiew 


Reuiew  Package 


Rep 

O 

IPT 

ther 

Reps 

C 

11 

-Reject 


Step  7 


Coordiiate  brmal 
claige  actbis  b 
tie  neqiinemeifc 
database. 


Accept- 


CUI 


■ 

START 

l 

FINISH 

-■W 


Reqiinemeife  Dative 


Impact  Analysis  Complete ... 
Submit  a  Change  Proposal 


Step  1 


Perform  Impai 
Aialyssaid 
cultimrat  witi 
IPT  Rep  ti  create 

CP(s). 


cultimrate 


Project 

Member 


Step  4  &  6 


Det  nniie  wlicICPsaid 
Siggesliois  t  neuiewaid 
asembte  reuiew  package/ 
CP  Let  DEtrfoit  actiois. 


Step  5 


CoidictCCB  ReuiewS 
DEpositni  of  CP  aid 
Siggesliois 


Step? 


Coordiiate  formal 
claige  actfois  t 
He  reqiiremeite 
database. 


Reuiew  Padcage 


IPT 


Rep 

O 

IPT 

ther 

Reps 

Cl 

Ul 

Reject 


Accept- 


-  Problem  Detected 


■ 

START 

1 

-Wf 


FINISH 


Reqiiremeite  Dative 


Submit  Change  Proposal 


Fill  out  appropriate  fields  in  the  ‘Proposed’  half  of  the  Change  proposal  Form.  Remember 
to  address  any  affected  attributes. 


|  Change  Proposal  for  module  'LAR'  -  DOORS 


Change  proposal  for  object:  LAR360 
Pending  change  proposals  for  this  object:  1 

IjCurrent-  - 

Object  Heading  | 

Object  T ext 


In-links:  0 
Out-links:  1 

Proposed 
Object  Heading  | 

Object  Text 


The  muzzle  brake  shall  not  generate  a  muzzle  exit  n 
pressure  above  12ksi. 


The  muzzle  brake  shall  not  generate  a  muzzle  blast 
overpressure  above  TBD.  (Driven  by  muzzle  exit 
pressure  of  12  ksi 


|atd 


~3 


ATD 


Make  adjustments  to  the 
Show  attribute:  |aTD/[  Reason  for  change  as  needed. 

BE  SURE  TO  NOTATE  ANY 
CONTRACTUAL 
IMPLICATIONS!!! 


Reason  for  change: 


fizzle  blast  overpressure  is  correct  term.  Muzzle  Brake  will  be  designed  to  minimize  blast  overpressure. 
Other  impactemequirements  are: 


1 


Change  type:  |  Modify  this  object  ~^\  Priority^  Medium 


~3 


Submit 


Select  Very  High,  ,  or 

(refer  to  CPP  Document  for  details) 


4 


When  satisfied  with 
form,  press  Submit  to 
create  the  new  Change 
proposal 


Submit  Change  Suggestion 


When  5  or  more  actions  need  to  occur  (I.e.,  Change  proposals)  in  order  to  fully  satisfy  a 
Change  Proposal,  a  Change  Suggestion  should  be  created  instead  of  a  change  proposal. 


Fill  out  fields  as  needed  and  press  Submit  to  create  a  new  suggestion.  The  JCCB  will 
approve  and  apply  suggestions  via  the  Change  Proposal  System. 


Review  CP’s  and  Suggestion 


Step  1 


Perform  Impact 
Aialyssaid 
cultimrat  witi 
IPT  Rep  b  create 

CP(s). 


col  borate  - 


Sit:m  rtC  laiqe 
Proposal  aid  or 
Suggestions! 
Sibmitadditbial 
CPsforinpactd 
objects. 


Step  4  &  6 


Det  nniie  which  CPs  aid 
Siggestbisbreuiewaid 
asembte  reiiiew  package/ 
CP  Lfit  DEtrtufc  actbis. 


Reuiew  Package 


Prd 

Met 

Uect 

nber 

IPT 

Rep 

O 

IPT 

ther 

Reps 

C 

U1 

-Reject 


Step  5 


CoidictCCB  ReuiewS 
DEpositni  of  CP  aid 
Siggestbis 


-Problem  Detected 


Step  7 


Coordiiate  formal 
claige  aotnis  t 
tie  reqiiremeite 
database. 


Accept- 


■ 

START 

i 

— 

- ■ 

FINISH 

-AfM 


Req  i  ireme  it  Daftse 


Predefined  Views  Can  Help 


|  Formal  module  7L5  Asmby/GA/LAR1  current  4.0  [RFP  12_6_01)  -  DOORS 


File  Edit  View  insert  Link  Analysis  Table  Tools  User  kitchen  New  Baseline  doorsconnect  forum  budgets  MFlAAS  Help 

I  H  #  iff  I  x  ^  e  |  y  |  x  v'  1 1*  i<  |  b  i  u  |  iff  ■-*,  |  m  T  =?*  m  | 

|  CP  Status  List 


IPj 


All  levels 


"31 


||j|gl|j|Ml  H  13  V  £1  I  m  w 


Object  Identifier  Launcher  Assembly  Requirements 


I  CP  Status  List 


M 


LAR37 


1  SCOPE 


LAR41 

LAR48 

LAR49 

LAR250 

LAR252 

LAR360 


3  REQUIREMENTS 

3.2  Characteristics 

3.2.1  Performance  Characteristics 

3.2. 1.9  Launcher  Assembly 

3.2. 1.9.1  Tube  Assembly 

The  muzzle  brake  shall  not  generate  a  muzzle  exit  pressure  above  12ksi. 


Views  can  be  built  in  an  RM 
Tool  to  help  in  the  review 
process . 


LAR50  3.2.2  Physical  Characteristics 
LAR334  3.2.2.4  Imbalance 

LAR335  The  Launcher  Assembly  shall  have  an  imbalance  of  no  more  than  1.0025  x  e7  N-mm  (7394  ft-lbs) 
total  Gun  Assembly  imbalance  is  equal  to  7457  ft-lhs.  Gun  Mount  is  63  ft-lbs.) 


d 


5  CP  LI -35 

Change  Type:  Modification 
Priority:  Medium 
Status:  New 

Reason  For  Change:  Muzzle  blast 
overpressure  is  correct  term.  Muzzle 
Brake  will  be  designed  to  minimize  blast 
overpressure. 

Other  impacted  requirements  are: 

GAR258:  The  Gun  Assembly  shall  not 
generate  a  muzzle  exit  pressure  above 
12ksi. 

MAR353:  The  Gun  Assembly  shall  not 
generate  a  muzzle  exit  pressure  above  12 
ksi. 

SYSR613:  The  maximum  muzzle  exit 
pressure  shall  not  exceed  12  ksi. 
Submitted  by:  alagasca 
Submitted  on:  27  February  2002 


(The  5  CP  LI  -34 

Change  Type:  Modification 
Priority:  Medium 
.Status:  In  Review 

Reason  For  Change:  Related  to  GP 
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Forms  Can  Also  Help 


ID  CP’s  and  Suggestions  and 

Schedule  JCCB 


Step  1 


Perform  Impact 
Aialyseaid 
cultimrat  witi 
IPT  Rep  b  create 

CP(s). 


col  borate  - 


Sit:m  rtC  laiqe 
Proposal  aid  or 
Suggestions 
Sibmitadditbial 
CPs  forinpacted 
objects. 


Step  5 


CoidictCCB  ReuiewS 
Dspositni  of  CP  aid 
Siggesliois 


Reuiew  Package 


Prd 

Met 

Uect 

nber 

IPT 

Rep 

O 

IPT 

ther 

Reps 

C 

U1 

-Reject 


-Problem  Detected 


Step  7 


Coordiiate  formal 
claige  aotnis  t 
tie  reqiiremeite 
database. 


Accept- 


■ 

START 

i 

— 

- ■ 

FINISH 

-AfM 


Req  i  ireme  it  Daftse 


Perform  JCCB  and  Update  dB  with 

Results. 


Approved  (ready  for  implementation) 
On-Hold  (further  investigation  needed) 
Rejected  (requested  change  discarded) 


Reaching  Consensus 


Use  IPT  forum  to  Elicit  Requirements. 

□  Include  Stakeholders  Early  and  Often. 

□  Have  Stakeholders  Peer  Review  Requirements 

□  Document  Rational.  It  will  save  you  time  latter 
when  you  will  need  to  defend  the  requirements. 

□  Use  a  JCCB 

□  Try  using  QFD  Method  to  Build  Consensus 


Communicating  Requirements 


Use  of  DOORS  has  helped  BUT!! 

□  Culture  shock  is  hard  to  overcome. 

□  Revert  back  to  WORD  and  EXCEL  documents. 

Not  so  efficient  and  may  introduce  errors. 

□  May  need  to  hold  hands 

□  Provide  Training  and  Tailor  it  to  the  project. 

□  Need  to  pay  close  attention  to  Permission  and 
database  administration  details. 

□  JCCB  has  forced  communication  to  happen  and 
has  made  it  mandatory. 

□  Will  need  good  IT  support  to  reach  remote 
locations  when  using  a  tool. 


Requirements  Metrics 


Select  metrics  you  will  use. 

Don’t  try  to  many  or  they  won’t  be  managed. 
You  can  build  them  into  an  RM  tool. 


Some  Examples  Include: 
Volatility 
#  Requirements 


#  TBD 

#  Verified 


Using  a  tool  will  produce 
metrics  naturally. 


Requirements  Attributes 


Attributes  are  additional  defined  characteristics 
of  a  requirement  and  they  provide  essential 
information  in  addition  to  requirement  text 


Source 

Priority 

Verifiability 

Accepted 

Review 

Safety 

Comments 

Questions 


Who  specified  this  requirement? 

What  is  the  priority  of  this  requirement? 

Is  the  requirement  verifiable? 

Has  this  requirement  been  accepted  by  the  developers? 
Review  status  of  this  requirement 
Is  this  a  safety-critical  requirement? 

Any  comments  on  the  requirement  to  clarify  its  meaning 
Any  questions  that  must  be  clarified  with  the  source 


You  can  define  attributes  that  will  support  your 
process  and  make  your  database  more 
productive  for  you 


Summary 


The  use  of  an  RM  tool  is  an  enabling  technology  to  achieve 
greater  accuracy  and  efficiency  when  engineering 
requirements. 

There  are  definite  skills  and  disciplines  required  to  do 
requirements  engineering 

Not  only  will  One  need  to  understand  how  to: 

□  Elicit  Requirements 

□  Capture  and  Control  Them 

□  Establish  and  maintain  Traceability 

□  Reach  Consensus 

□  Elicit  Verification  Methods 

□  Communicate  Requirements 

□  Defined  some  Metrics  and  Attributes 

They  will  also  need  to  be  proficient  in  using  and  tailoring  an  RM 
Tool 


Questions? 


NDIA  11th  Annual  Systems  Engineering 

Conference 


“ The  Value  of  Architecture” 

October,  2008 


Frank  Salvatore 

High  Performance  Technologies,  inc. 

3159  Schrader  Road 

Dover  NJ,  07801 

(973)  442-6436  ext  249 

fsalvatore@hpti.com 


Outline 


□  Architecture 

□  Operational  View 

□  Goal  Hierarchy 

□  Process  Flow 

□  7.0  Identify  and  Define  Alternatives 

□  Tools  Architecture 

□  Summary 


Architecture 


□  During  the  systems  engineering  process 
architectures  are  generated  to  better  describe 
and  understand  the  system 

□  Architectures  provide  a  description  of  how 
subsystems  join  together  to  form  a  system. 

■  The  Functional  Architecture  identifies  and  structures  the 
allocated  functional  and  performance  requirements. 

■  The  Physical  Architecture  depicts  the  system  product  by 
showing  how  it  is  broken  down  into  subsystems  and 
components. 

■  The  System  Architecture  identifies  all  the  products 
(including  enabling  products)  that  are  necessary 

■  Operational  Views  provide  a  frame  of  reference  that  the 
project  work  can  be  related  to. 


Operational  View 


Identify,  define,  and  evaluate  potential  Universal  (Objective) 
Active  Protection  System  (APS)  approaches  for  the  Future 
Combat  System  (FCS). 


Provide  decision  makers  the  tools/data  to  help  identify 
RDECOM’s  Science  and  Technology  investments  needed  to 
get  to  an  objective  APS  system. 


An  Operational  View  was  key.  It  gave  everyone  a  common  frame  of 
reference  to  work  from  when  executing  their  part  of  the  analysis. 


Goal  Hierarchy 


Determ  in  e  Criti  ca  I  APS 
T  echnologiesy  Architecture 
for  FCS  Success 


I 

Goal  1  Maximize 
Protection  of 
Platform 


—  1.1 


Goal  2  Maximize 
Fratricide 
Control 


2,1 


Goal  3  Maximize 
Operability 


—  1.2  —  2,2  —  3,2  —  4.2 


I—  1.3  —  2.3  I— 


3.1 


Goal  4  Minimize 
Interface  Needs 


—  4.1 


3.3 


l—  2.4 


4.3 


4.4 


Goal  5  Minimize  < 

Schedule  Risk 

Goal  6  Minimize 

Cost 

5.1 

— 

6.1 

5.2 

— 

6.2 

5.3 

— 

6.3 

This  was  the  Goal  Hierarchy.  Essentially  an  Arhcitecture.  Without  it  we 
were  not  focused  on  what  was  important  to  consider  in  the  trade  study 


Process  Flow 


Trade  Study  Process  Flow  Diagram  was  the  Process  Architecture 
used.  It  kept  the  team  aligned  and  was  a  central  communication  tool 


7.0  Identify  &  Define  Alternatives 


•  List  Systems/Components 

•  Previous  Trades  •  Existing  Systems 

•  Component  Data  •  Analysis  Method, Tools  •  System  Alternatives 

•  Requirements  •  System  Assumptions  •  System  ID 


•  Integrate  System  Candidates  ^Analyze  System  Candidate  Potential 

•  Organize  Component  Data  •  Timeline 

•  ID  Functional  Architectures  •  Accuracy 

•  Component  Compatibility 


System  and  Technology  Architectures  Required!!!!! 


7. 1  Candidate  Systems 
(Physical  Architecture) 


The  Physical  Architecture  was  core  to  understanding  the  basic  construct  of  an  Active 
Protection  System.  All  10,080  Systems  Evaluate  had  the  same  Physical  Architectures 


7.2  Evaluate  Candidates 
(Functional  Analysis  and  Allocation) 


Major  component  of  the  trade  study  was  the  Functional 
Analysis  and  Allocation  (FAA). 

■  It  allowed  for  a  better  understanding  of  what  the  technologies 
could  and  had  to  be  able  to  do  to  satisfy  the  performance 
requirements  of  the  system,  in  what  ways  they  could  do  it,  and 
to  some  extent,  the  priorities  and  conflicts  associated  with 
lower-level  functions. 


■  It  provided  information  essential  to  optimizing  physical  solutions. 

■  Key  tools  were  Functional  Flow  Block  Diagrams,  and  the  Time 

Line  Analysis  .Ust  Systems/Components 

•  Previous  Trades  •  Existing  Systems 

•  Component  Data  •  Analysis  Method.Tools  •  System  Alternatives 

•  Requirements  •  System  Assumptions  •  System  ID 


•  Integrate  System  Candidates  "Analyze  System  Candidate  Potential 

•  Organize  Component  Data  ■  Timeline 

•  ID  Functional  Architectures  •  Accuracy 

•  Component  Compatibility 


7.2  Evaluate  Candidates 
(System  Functions) 


Function 

Definition 

Detect,  Acquire 

Measure  and  report  an  event  not  due  to  ambient  noise 

Declare 

Measure  and  report  an  persistent  object  that  should  be  tracked 

Classify 

Measure  and  report  what  the  persistent  object  is  either  by  class  or  specific 
type/item. 

Coarse  Track 

Measure  and  report  an  object  and  determine  that  it’s  trajectory  point  of  closest 
approach  to  our  platform  is  threatening.  Classify  and  coarse  track  may  be 
based  on  the  same  measured  data  set  and  completed  at  the  same  time 

Initial  Slew 

Initial  slew  of  launcher  to  launch  position  using  fire  control  solution  based  on 
coarse  track 

Initial  Tube  Selection 

Initial  designation  of  launch  tube  or  tubes  in  fixed  system  that  need  to  be 
“warmed  up”  using  fire  control  solution  based  on  coarse  track 

Fine  Track 

Measure  and  report  a  target  to  enable  calculation  of  a  fire  control  solution 

Fine  Slew  &  Fire  Control 

Slew  launcher  to  final  position  and  launch  an  interceptor  loaded  with  any 
required  flight  path,  terminal  guidance,  and  fuzing  information 

Final  Tube  Selection  &  Fire 
Control 

Final  designation  of  launch  tube  in  fixed  system  and  launch  an  interceptor 
loaded  with  any  required  flight  path,  terminal  guidance,  and  fuzing  information 

Established  a  common  vocabulary  for  understanding  and  describing  how  each  for  the 

systems  studies  operated. 


7.2  Evaluate  Candidates 
System  Functions  (cont.) 


Function 

Definition 

In-Flight  Track 

Measure  and  report  a  target  trajectory  to  provide  in-flight  guidance  to  an 
interceptor 

No-Op 

“No  operation”  -  used  to  designate  function  not  performed 

In-Flight  Guidance 

Propulsion  to  change  flight  path  of  interceptor 

Terminal  Track 

Measure  and  report  a  target  trajectory  to  provide  terminal  guidance  &  fuzing 
updates  to  an  interceptor 

Terminal  Guidance  &  Fuze 

Orient  (focus)  the  warhead  to  produce  the  desired  effect  &  initiate  the  effect  at 
the  prescribed  time  and  /  or  the  prescribed  distance  from  target 

Warhead  Effect 

Target  negation 

Established  a  common  vocabulary  for  understanding  and  describing  how  each  for  the 

systems  studies  operated. 


7.2  Evaluate  Candidates 

Functional  Flow  Block  Diagram  (Unguided  Interceptor) 


7.2  Evaluate  Candidates 

Functional  Flow  Block  Diagram  (Guided  Interceptor) 


7.2  Evaluate  Candidates 
(Functional  to  Physical  Allocation) 


Architectures  for  Unguided  Interceptors 

Architectures  for  Guided  Interceptors 

U1 

U2 

U3 

U4 

G1 

G2 

G3 

G4 

System 

Functions 

Detect,  Acquire  &  Declare 

Passive  Cuer 

Passive  Cuer 
/  Coarse 
Tracker 

Passive  Cuer 

Active  Cuer  / 
Tracker 

Passive  Cuer 

Passive  Cuer 
/  Coarse 
Tracker 

Passive  Cuer 

Active  Cuer  / 
Tracker 

Classify 

Active  Tracker 

Passive  or 
Active  Coarse 
Tracker 

Active  Tracker 

Passive  or 
Active  Coarse 
Tracker 

Coarse  Track 

Initial  Slew  /  Tube 

Selection 

Launcher 

Launcher 

Launcher 

Launcher 

Launcher 

Launcher 

Launcher 

Launcher 

Fine  Track 

Active  Tracker 

Active  Fine 
Tracker 

Active  Fine 
Tracker 

Active  Cuer  / 
Tracker 

Active  Tracker 

Active  Fine 
Tracker 

Active  Fine 
Tracker 

Active  Cuer  / 
Tracker 

Final  Slew  /  Tube  Selection 
&  Fire  Control 

Launcher 

Launcher 

Launcher 

Launcher 

Launcher 

Launcher 

Launcher 

Launcher 

In-Flight  Track 

None 

None 

None 

None 

Active  Tracker 

Active  Fine 
Tracker 

Active  Fine 
Tracker 

Active  Cuer  / 
Tracker 

In-Flight  Guidance 

Guided 

Interceptor 

Guided 

Interceptor 

Guided 

Interceptor 

Guided 

Interceptor 

Terminal  Track 

Unguided 

Interceptor 

Unguided 

Interceptor 

Unguided 

Interceptor 

Unguided 

Interceptor 

Active  Tracker 

Active  Fine 
Tracker 

Active  Fine 
Tracker 

Active  Cuer  / 
Tracker 

Terminal  Guidance  &  Fuze 

Guided 

Interceptor 

Guided 

Interceptor 

Guided 

Interceptor 

Guided 

Interceptor 

Warhead  Effect 

Functional  allocation  to  physical  components  provided  context  for  data  provided  on 
specific  components  and  was  critical  in  both  the  Timeline  and  Accuracy  Analysis. 


Threat 


7.2  Evaluate  Candidates 
Timeline  Analysis 
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Pass  Screen 


The  results  of  the  Functional  Analysis  and  Allocation  effort  provided  the  basis  for  how  time  was 
to  be  calculated  for  each  of  the  10K  plus  systems  to  be  evaluated. 


7.2  Evaluate  Candidates 
Interface  Compatibility  Analysis 


SCORING  INSTRUCTIONS 


1  Level  1 

3 

1  Component  Compatibility  Description 

-  Significant  software  integration  with  concurrently  developed  hardware. 

-  Hardware  and/or  software  interfaces  defined  and  analyzed  so  complexity  is 

1 

-  Software  and/or  hardware  interfaces  known  but  need  to  be  revised  with  as 

0 

-  Interfaces  exist  and  no  changes  are  required. 

Hardware  interface  c 

-  Mechanical  -  envelope,  attachment,  obscuration,  alignment 

-  Hydraulic  and  pneumatic  -  flow  rates,  pressures 

-  Mass  -  weight,  moments  of  inertia,  centers  of  gravity 

-  Environment  -  mechanical  shock  and  vibration,  particulate,  elf 

-  Thermal  -  temperature  limits,  temperature  control 

-  Electrical  -  signals,  voltage,  power 

Software  interface  considerations  include  added  requirements  for 

-  Data  encryption  and  encoding 

-  Data  structures 

-  Data  storage 

-  Data  transfer  rates 

-  Data  communication  protocols 

-  Data  processing  and  algorithms 


Experties 


0 

No  experties,  Don't  fill  out  scores  for  anything  you  have  no  exp 

1 

If  you  have  seen  a  briefing  on  the  technology  or  have  only  rece 

3 

If  you  have  a  working  knowledge  (understand  underlying  physic 

9 

If  you  are  intimately  involved  in  designing,  developing,  and  or  in 

Cue  -  Track  Results 
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Results 
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Physical  to  Functional  Allocations  helped  in  determining  what  the  interfaces  would  be  and  gave  us 

a  way  to  make  subjective  evaluations  of  their  maturity 


7.3  Define  Alternatives 


7.2  Evaluate  Candidates 
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8.0  Evaluate  /  Score  Alternatives 


Physical  to  Functional  Allocation  allowed  us  to  define  the  system  configuration,  system 
architecture,  and  principle  of  operation  of  each  system  analyzed. 


Abstract  Architecture 

□  Schematic  Block  Diagrams 

■  Physical  Architecture 

■  Interfaces 

■  Data  Flow 

■  Easy  to  Read 

■  Hard  to  Maintain 


Formal  Architecture 

□  IDEFO,  FFBD,  EFFBD,  Hierarchy 

■  Physical  Architecture 

■  Functional  Architecture 

■  Interfaces 

■  Data  Flow 

■  Easy  to  Maintain 

■  Hard  to  Read 


Tools  Architecture 
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[Threat  Community]  [S&T  Community] 


Perform  APS  Analysis 
Functional  Flow  Block  Diagram  (FFBD) 


Hierarchy  Diagram 
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The  Hierarchy  Diagram  was  a  quick  way  to  quickly  capture  all  the  Trade  Study  Tools 
and  their  Hierarchical  relationships.  These  ultimately  became  the  configuration  items 

that  were  kept  under  version  control. 


Summary 


□  Use  of  Business  Process  Models  helped  everyone  to  understand  the  trade 
study  approach  that  was  being  used. 

□  Using  Hierarchy  Diagrams  helped  the  trade  study  team  stay  focused  on  the 
goals  and  criteria  being  evaluated. 

□  Physical  Architecture,  Functional  Architectures  provided  the  trade  study  team 
and  the  rest  of  industry  a  common  language  to  work  from.  It  also  was  core  to 
defining  systems,  organizing  data 

□  Functional  Flow  Block  Diagrams  and  Functional  To  Physical  Allocation  was 
instrumental  to  establishing  rules  used  to  automating  the  evaluation  of  10K  plus 
system  alternatives.  More  importantly  it  allowed  the  entire  APS  community  to 
agree  it  was  being  done  correctly  in  all  10k  plus  cases. 

□  Capturing  System  Architectures  was  essential  to  understand  how  to  model 
system  time  function  and  communicate  it  to  the  community. 

□  Structured  Physical  and  Functional  decomposition  made  establishing  a  System 
ID  scheme  simple. 

□  Tool  Architecture  helped  to  communicate  how  each  tool  was  used  in  the  trade 
study  process 

many  tool  interface  gaps  were  identified  and  fixed. 
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Improving  Weapon  System 
Investment  Decisions 


A  Knowledge-based  Approach  to  Weapon 
System  Acquisitions  Could  Improve  Outcomes 


Travis  Masters 
Senior  Defense  Analyst 
U.S.  Government  Accountability  Office 
Wright-Patterson  AFB,  OH 
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DOD  Framework  vs.  Knowledge-based 
Best  Practices  Model 


DOD"s  Framework 
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Major  Determinant  Of  Program  Outcomes  Is  The 
Level  Of  Knowledge  Attained  At  Key  Junctures 

Knowledge  Point  1:  At  milestone  B,  a  match  is  achieved 
between  the  user’s  needs  and  the  developer’s  resources 
(indicator:  technology  readiness  level) 

Knowledge  Point  2:  At  critical  design  review,  the  product 
design  demonstrates  its  ability  to  meet  user  needs  and  is 
stable  (indicator:  %  of  engineering  drawings  released) 

Knowledge  Point  3:  At  milestone  C,  it  is  demonstrated  that  the 
product  can  be  produced  within  cost,  schedule,  and  quality 
targets  (indicator:  %  of  key  processes  in  statistical  control) 


A  GAO 
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Making  a  Business  Case  that  a  Product  Can 
Be  Developed  Within  Resource  Constraints 


At  milestone  B  programs  should  present  a  business  case 
that  provides  evidence  that: 


(l)Warfighter  needs  are  valid  and  can  be  met  with 
chosen  concept,  and 


(2)The  chosen  concept  can  be  developed  and 
produced  within  resources-technologies,  funding, 
design  knowledge,  and  time 
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Resolving  Gaps  Between  Requirements  and 
Resources  Before  Program  Start 

Early  systems  engineering  enables  a  developer  to  identify 
and  resolve  gaps  between  resources  and  requirements 
before  product  development  begins 


Source:  GAO. 
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DOD  Programs  Continue  to  Experience  Cost 
and  Schedule  Problems 


50% 


Change  in  Total  Change  in  RDT&E  Share  of  programs  with 

Acquisition  Costs  From  Costs  From  First  25  percent  cost  growth 

Fi  rst  Esti  mates  Esti  mates 


□  FY  2000  Portfolio  □  FY  2005  Portfolio  ■  FY  2007  Portfolio 


Months 


■  FY  2000  Portfolio  □  FY2005  Portfolio  ■  FY2007  Portfolio 


Source:  GAO  analysis  of  DOD  data. 
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GAO  Continues  to  Find  That  Programs  Begin 
Without  Key  Knowledge 

•  Requirements  are  not  well  understood 

•  Quantum  leaps  in  capability  not  incremental  changes 

•  Technologies  are  not  mature 

•  Cost  and  schedule  estimates  are  overly  optimistic 

•  Program  cycle  times  are  too  lengthy 
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Little  Evidence  of  Widespread  Adoption  of 
Knowledge-based  Acquisition  Process 


DOD’s  acquisition  practices  necessary  to  ensure  effective  implementation  of 
knowledge-based  process  are  not  always  followed  despite  policies  and  guidance  to 
the  contrary. 


Key 

junctures  Development  start  Design  review  Production  start 


Knowledge  point  1 


Knowledge  point  2 


Knowledge  point  3 


Best  Mature  all  critical 

practices  technologies 


Achieve  knowledge  point 
1  on  time  and  complete 
90  percent  of  engineering 
drawings 


Achieve  knowledge  points 
1  and  2  on  time,  and  have  all 
critical  processes  under 
statistical  control 


DOD 

outcomes3 


12  percent 
of  programs 


4  percent  of 
programs 


0  percent  of 
programs13 


Source:  GAO  presentation  of  DOD  data. 

a  Not  all  programs  provided  information  for  each  knowledge  point  or  had  passed  through  all  three  key  junctures, 
b  In  our  assessment  of  two  programs,  the  Light  Utility  Helicopter  and  the  Joint  Cargo  Aircraft,  are  depicted  as  meeting  all  three 
knowledge  points  when  they  began  at  production  start.  We  excluded  these  two  programs  from  our  analysis  because  they  were 
based  on  commercially  available  products  and  we  did  not  assess  their  knowledge  attainment  with  our  best  practices  metrics. 
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GAO’s  Review  of  the  Acquisition  Decision 
Support  Systems 


GAO  has  done  a  lot  of 
work  looking  at  the  DAS 


Congress  directed  GAO 
to  initiate  a  body  of  work 
looking  at  the  funding  and 
requirements  processes 
and  how  they  could 
support  better  program 
outcomes 


k  GAO 
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Portfolio  Management:  A  Successful 
Commercial  Model 


•  Each  investment  must  be  viewed  from  an  enterprise  level  as 
contributing  to  the  collective  whole,  rather  than  independent 
and  unrelated 

•  Identify  and  Prioritize  Market  Opportunities  to  Lay  the 
Foundation  for  Achieving  the  Right  Mix  of  Products 


•  Use  a  Disciplined  Process  to  Identify  New  Products  and 
Achieve  a  Balanced  Portfolio 

•  Ensure  strong  governance,  committed  leadership,  empowered 
decision  makers,  and  effective  accountability 


GAO-07-388 
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The  Portfolio  Management  Funnel 


SoLrce:G££i  anelyslBand  pewnlaacri  ol  commercial  pracflcee. 


DOD’s  Decision  Making  Processes  are 
Service-centric  and  Fragmented 
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•  Services  identify  needs  and  budget  for  solutions 

•  FCBs  don’t  have  the  resources  to  effectively  evaluate  the  service 
assessments  within  the  context  of  the  broader  portfolio 

•  FCBs  don’t  have  the  authority  to  allocate  resources 

•  Service  funding  appears  to  be  allocated  according  to 
historical  percentages 

•  40%  AF,  20%  Army,  30%  Navy,  and  10%  DOD  Wide 


JCIDS,  PPBE,  and  DAS  led  by  different  organizations 

•  Joint  Staff,  USD(AT&L),  OSD  (PA&E  and  Comptroller) 
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Fragmented  Processes  With  Adverse 
Incentives 


Source:  GAO. 


PRESSURE  ON 
DECISION  MAKER  TO... 

...  promise  high 
performance 


...  promise  low 
resource  demands 


...  move  forward, 
get  knowledge  later 
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DOD  Commits  to  Solutions  Early  and  With 
Limited  Knowledge 

•  Review  points  prior  to  milestone  B  are  “optional”  and 
typically  by-passed 


•  Key  processes  are  not  integrated  early  to  provide  insight 
into  cost  and  feasibility 

•  ICDs  don’t  address  cost  or  technical  feasibility 

•  AOAs  often  make  the  case  for  a  specific  solution  vs.  identifying  the 
preferred  solution 


•  Programs  don’t  have  sound  business  cases 

•  Undefined  requirements 

•  Immature  technology 

•  Optimistic  cost  and  schedule  estimates 


AGAO 
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DOD’s  Funding  Process  Contributes  to  Poor 
Acquisition  Outcomes 


•  Assessed  cost  and  funding  data  for  20  major  acquisition  programs, 
and  conducted  detailed  analysis  of  five  of  those  programs: 

•  Global  Hawk 

•  Joint  Strike  Fighter  (JSF) 

•  Future  Combat  Systems  (FCS) 

•  Warfighter  Information  Network  -  Tactical  (WIN-T) 

•  Multi-mission  Maritime  Aircraft  (MMA) 

•  Leveraged  work  GAO  has  been  doing  in  cost  estimating  and 
earned  value  best  practices  ( GAO  Cost  Assessment  Guidebook) 


•  Leveraged  prior  best  practices  work  and  obtained  additional  input 
from  several  of  the  companies  that  contributed  to  our  prior  work 


GAO-08-619 
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Accurate  Cost  Estimates  Are  Needed  Before 
Adequate  Funding  Can  Be  Allocated 

•  Without  accurate  estimates  it  is  not  realistic  to  assume 
that  funding  will  be  adequate 


•  Cost  estimating  best  practice  is  to  assess  risk  and 
uncertainty  and  present  estimate  as  range  of  potential 
costs 

•  Conduct  sensitivity  analysis  and  identify  the  range  of  likely  costs 

•  Ranges  will  be  “broader”  as  knowledge  is  limited  but  as  knowledge  is 
gained  (before  development  begins)  the  range  should  “narrow”  until 

•  Ranges  allow  decision  makers  to  make  more  informed  decisions — 
they  can  test  the  estimate’s  reasonableness  and  decide  on  what  level 
of  funding  risk  they  want  to  take) 
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The  Cone  of  Uncertainty 
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Built-in  Funding  Instability 


•  DOD  programs  often  initiate  development  with 
funding  that  does  not  reflect  true  costs 

•  75%  of  the  programs  we  reviewed  were  under-funded  in  the 
FYDP  when  they  began  development 

•  The  FYDP  doesn’t  cover  the  entire  development  program 


•  DOD  makes  unplanned  and  inefficient  adjustments 
to  compensate  for  poor  planning  /  projections 

•  Creates  /  perpetuates  instability 

•  Pushes  costs  into  the  future 

•  “Robs  Peter  to  pay  Paul” 

•  Reduces  procurement  quantities 
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Unrealistic  Cost  Estimates  Hinder  Accurate 
Funding  Commitments 


•  Estimates  are  often  based  on  limited  knowledge  about 
requirements  and  technologies  and  optimistic  assumptions — 
lack  of  systems  engineering  analysis  up  front 


•  Our  analysis  of  20  programs  found  that  both  CAIG  and 
Service  estimates  tended  to  be  too  low 


•  Estimates  are  presented  as  point  estimates  representing 
“most  likely  cost”  and  do  not  depict  risk  and  uncertainty 


•  Program  cycle  times  are  longer  than  the  FYDP  timeframe 
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DOD’s  Failure  to  Balance  Needs  with 
Resources  Promotes  Unhealthy  Competition 

•  Relying  on  unrealistically  low  estimates,  DOD  has  committed 
to  more  programs  than  its  resources  can  support 


•  In  a  zero-sum  game,  increases  in  one  program  will  impact 
other  programs 


•  Pressure  to  make  a  program  stand  out  from  others 


•  Pressure  to  appear  affordable  (fit  within  the  FYDP) 


•  When  “reality”  hits  and  things  don’t  go  as  planned,  instability 
is  the  inevitable  result 


Accountability  *  Integrity  *  Reliability 


Recommended  Steps  to  Improve  Program 
Funding 


•  Balance  the  current  portfolio  (to  reduce  the  pressures  of 
unhealthy  competition) 


•  Require  programs  to  have  short,  manageable  development 
cycles  (5  to  6  years  long) 


•  Require  cost  estimates  to  be  presented  as  a  range  of  likely 
costs  (wider  at  a  milestone  A  point  and  more  narrow  at 
milestone  B) 
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DOD’s  Requirements  Process  (JCIDS)  Has  Not 
Been  Effective  in  Prioritizing  Joint  Capabilities 


•  JCIDS  is  not  meeting  its  objective  to  prioritize  joint  warfighting 
needs 

•  Military  services,  not  the  joint  warfighting  community  continue  to  sponsor 
most  JCIDS  proposals 

•  Almost  70%  of  initial  capability  proposals  submitted  to  JCIDS  since 
2003  were  sponsored  by  a  military  service 

•  Virtually  all  capability  proposals  that  go  through  the  JCIDS  process  are 
validated — or  approved 

•  Of  140  capability  proposals  since  2003  that  completed  the  process, 
only  6  were  not  validated 

•  Process  is  also  lengthy  and  cumbersome,  making  it  difficult  to  respond  to 
near-term  needs 


•  DOD  is  losing  opportunities  to  strengthen  joint  warfighting 

capabilities  and  constrain  its  portfolio  of  weapon  system  programs 
to  match  available  resources 


GAO-08- 1060 
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DOD  Lacks  An  Approach  and  Alignment  of  Resources 
to  Prioritize  and  Balance  Capability  Needs 


•  JCIDS  largely  responds  to  capability  proposals  that  are 
submitted  by  sponsors  on  a  case-by-case  basis 

•  Lacking  a  more  proactive  and  analytic  approach,  JCIDS  has 
been  ineffective  at  integrating  and  balancing  needs 

•  The  military  services  continue  to  drive  the  determination  of 
capability  needs,  in  part  because  they  retain  most  of  DOD’s 
analytic  capacity  and  resources 

•  Without  an  approach  and  entity  in  charge  to  determine  what 
capabilities  are  needed,  all  proposals  tend  to  be  treated  as 
priorities  within  the  JCIDS  process 


GAO 
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Recommended  Steps  To  Improve  JCIDS 


•  Develop  an  analytic  approach  within  JCIDS  to 
better  prioritize  and  balance  capability  needs 
department-wide,  and 


•  Determine  and  allocate  appropriate  resources 
for  conducting  joint  capabilities  development 
planning 
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VV&A  -  where  it  all  started  ... 

"Those  who  cannot  remember  the  past  are  condemned  to  repeat  it. " 


By  the  Comptroller  General  Report  to  the  Congress  of  the  Unites  States, 
Models,  Data,  and  War:  A  Critique  of  the  Foundation  for  Defense 
Analyses,  March  12,  1980  (PAD-80-21) 

(http://archive.gao.gOv/f0202/1 1 1782.pdf) 

United  States  General  Accounting  Office  Report  to  the  Chairman, 
Legislation  and  national  Security  Subcommittee,  Committee  on 
Government  Operations,  House  of  Representatives.  DoD  Simulations: 
Improved  Assessment  Procedures  Would  Increase  the  Credibility  of 
Results  (http://archive.gao.gov/d30t5/1 34959.pdf) 

Military  Operational  Research  Society  (MORS)  Simulation  Validation 
Mini-Symposiums  and  Workshops  (http://www.mors.org/reports.htm) 

Report  of  the  Defense  Science  Board  Task  Force  on  Simulation, 
Readiness  and  Prototyping.  Impact  of  Advanced  Distributed  Simulation 
on  Readiness,  Training  and  Prototyping,  January  1993 
(http://www.acq.osd.mil/dsb/reports/srp.pdf) 

DoDD  5000.59  DoD  M&S  Management,  January  4,  1994 

DoDI  5000.61  DoD  M&S  VV&A,  April  29,  1996 
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Everything  Is  Simulation  Except  Combat* * 


•  Modeling  and  Simulation 
(M&S)  is  a  key  enabler  for 
systems  engineers  in  the 
acquisition  process 

•  Using  M&S  that  provide 
credible  results  is  crucial  to 
fielding  defense  weapon 
systems  to  the  warfighter 

•  Credibility  and  confidence  in 
the  use  of  M&S  results  are 
achieved  through 
implementation  of  Verification, 
Validation,  and  Accreditation 
(VV&A)  processes 

•  VV&A  is  critical  for  ensuring 
M&S  is  correct,  is  used 
correctly,  and  can  produce 
results  a  systems  engineer 
can  trust 


Naval  Undersea  Warfare  Center  Newport 
Synthetic  Environment  Tactical  Integration 
Virtual  Torpedo  Project 

HLA  federation  linking  live  submarines  to  high-fidelity 
torpedo  hardware-in-the-loop  facility 


*Defense  Science  Board,  January  1993 
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Three  Methods  of  Simulation 

‘Defense  Science  Board,  January  1993 


*Findings  -  Continued 
DoD  investment  required 
in  VV&A:  “Techniques 
routinely  used  for  VV&A  of 
single  models  or  simulations 
face  new  challenges  in  a 
multi-source,  highly 
interactive,  internetted  M&S 
environment  where  complex 
software  modules  are 
required  to  interoperate. 

New  techniques  of  VV&A 
are  likely  required.  ” 

“The  important  task  of 
verifying,  validating,  and 
accrediting  battlefield 
behavior,  modeled  in  some 
form,  should  receive 
greater  attention  in  all  DoD 
M&S  programs.  ” 


LIVE 


Real  people  operating  real 
systems  in  real  environments  in 
the  air  and  space,  on  the  ground, 
on  and  below  the  sea 


VIRTUAL 

Real  people  operating  simulated 
systems  in  simulated  environments 
included  are  wargames,  models  and 
analytic  tools 


CONSTRUCTIVE 

Simulated  people  operating 
simulated  systems  in  simulated 
environments 


DoD  M&S  Project 


•  Project  title:  Standardized  Documentation 

for  VV&A 


•  Sponsor:  Department  of  Defense  (DoD) 

M&S  Steering  Committee 
(M&S  SC) 


•  Oversight: 


Acquisition 

Community 

Lead 


OOUSD,  ^cquisrf  non  &  Technology 
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Project  Organization 


M&S 

Coordination 
Office 
(M&S  CO) 
Acquisition 
Community 
Coordinator 


M&S  CO  '■  I 
W&A  ;  ' 


Project  Scope 


•  Three  major  tasks  and  associated 
deliverables: 

-  recommend  updates  to  associated  policy, 
guidance,  and  standards  documents 

-  design,  describe,  and  register  VV&A  XML 
schemas 

-  design,  develop,  test,  and  deploy  the  DVDT 
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Plan  of  Action  and  Milestones 
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Concept  of  Operations 


Policy,  Guidance  &  Standards 


\L 


Level  1 


M&S 

DoDI  5000.59 


Acquisition 
DoDI  5000.2 


Data 

i 

DoD  Discovery 
Metadata 

Specification  (DDMS) 


Level  2 


MIL 


-STD-3022  m 


DoDI  5000.61 


Defense  Acquisition  Guidebook 
•Sec 4.5.7  M&S 
•Sec  9.6.2  &  9.10  TEMP 
- 1  - 


M&S  Community  of 
Interest  Discovery 
Metadata  Specification 

(MSC  DMS)  _ s£\ 


DoD  VV&A 

DAU  Continuous  Learning  Modules 

Recommended  A 

•  CLE011  M&S  in  Sys  Eng 

Practices  Guide^^ 

•  CLE  023  M&S  for  T&E 

MIL-STD-3022 
28  January  2008 


•  2008:  Approved  as  a  DoD  Standard 
Practice  with  four  associated  Data  Item 
Descriptions  (DIDs) 

-  DI-MSSM-81750  DoD  M&S  Accreditation 
Plan 

-  DI-MSSM-81751  DoD  M&S  V&V  Plan 

-  DI-MSSM-81752  DoD  M&S  V&V  Report 

-  DI-MSSM-81753  DoD  M&S  Accreditation 
Report 

•  MIL-STD-3022  may  be  cited  as  a 
solicitation  requirement  and  DIDs 
included  on  Contract  Data  Requirements 
List 

•  Available  at  Acquisition  Streamlining  and 
Standardization  Information  System 
(ASSIST)  -  http://assist.daps.dla.mil/ 


NOT  MEASUREMENT 
SENSITIVE 


Ml  L-STD-302  2 
2£  January  2O0& 


DEPARTMENT  OP  DEFENSE 
STANDARD  PRACTICE 

DOCUMENTATION  OF 
VERIFICATION.  V  ALIDATION.  AND 
ACCREDITATION  (VV&A) 

FOR  MODELS  AND  SIMULATIONS 


AMSC:  9037 


AREA:  M53M 


DVDT  automates  standard 
templates  enabling  sharing  of  VV&A 
information  across  Global 
Information  Grid  (GIG)  enterprise 
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DoD  Instruction  5000.61 
M&SVV&A,  13  May  2003 

•  M&S  SC  directed  a  5-year  review 
in  2008 

•  Working  Group  kickoff  meeting  (21 
Feb  2008) 

-  Drafting  Group  formed 

•  M&S  IPT  informal  review  of  draft 
revision  (25  Jul-8  Aug  2008) 

•  Comment  Resolution  Panel  (CRP) 

(Aug-Sep  2008) 

•  USD(AT&L)  Review  Process 

-  Not  begun 

•  DoD  Directives  Program 
Coordination  Process  (SD-106) 

-  Signature  authorities  for  DoD  issuances 
include  Presidentially  Appointed, 

Senate-confirmed  (PAS)  officials 

-  Processing  might  wait  for  new 
administration 


DoD  SD-106  Process 
USD(AT & L)  Process 
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Discovery  Metadata 


DoD  Discovery 
Metadata 
Specification 
(DDMS)  describes 
DoD  resources 
-  is  very  broad 


M&S  Community  of 
Interest  Discovery 
Metadata  Specification 
(MSC  DMS)  provides 
more  detail  to  describe 
resources  in  the  M&S 
domain 

-  is  more  precise 


VV&A  metadata  describes 
VV&A  resources 

-  is  most  precise 


VV&A 


GIG  Enterprise  Services 
Metadata  Working  Group 


GIG  M&S 
Community 
of  Interest 
(COI) 


DoD  M&S 
Project 
Standardized 
Documentation 
for  VV&A 
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DoD  Metadata  Registry 


W&A 

Project 

Metadata 

XML 

Schema 


Accreditation 
Plan 
XML 
Schema 


V&V 
Plan 
XML 
Schema 


W&A 


Document 

Base 

Types 

XML 


Schema 


Accreditation 

Report 

XML 

Schema 

V&V 

Report 

XML 

Schema 

The  DoD  Metadata  Registry  and 
metadata  registration  process 
together  collect,  store,  and 
disseminate  structural  metadata 
information  resources,  e.g.: 

•  schemas 

•  data  elements 

•  attributes 

•  document  type  definitions 

•  style-sheets 

•  data  structures 

The  project’s  XML  products  will  be 
registered  and  available  for  use  by 
industry  and  government. 


http://metadata.dod.mil 
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5 


DVDT 


•  Automates  production  of  VV&A  documentation  in 
compliance  with  MIL-STD-3022 

•  Enables  search  and  discovery  of  VV&A  document 
information  via  the  GIG  enterprise 

•  Produces  documents  that  enable  DoD  data  sharing  in 
compliance  with: 

-  DoD  Directive  8320.02,  Data  Sharing  in  a  Net-Centric  DoD 

-  DoD  Discovery  Metadata  Specification 

-  M&S  Community  of  Interest  Discovery  Metadata  Specification 

•  Helps  organizations 

-  produce  documents  more  efficiently  and  consistently 

-  organize  information 

-  output  documents  in  a  common  format 

-  share  discovery  metadata  about  VV&A  documents  across  GIG 
enterprise 
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Architecture  Focus 


•  In  January  2008  focus  turned 


-  to  an  architecture  that  would  allow  offline  document  production  and 
storage 

-  from  one  that  would  provide  an  online  capability  to  produce 
documents 


•  Offline  production  and  storage  capability  assists 
producer  in  managing  information  common  to  all  four 
documents  identified  in  MIL-STD-3022 


•  DVDT  populates  the  four  documents  with  the 
common  information  when  documents  are  stored 
together 

•  Coordination  on  production  of  each  individual 
document  by  disparate  organizations  will  occur 
through  means  used  by  those  organizations 

-  e.g.,  integrated  digital  environment,  engineering  environment, 
knowledge  sharing  environment,  or  sharing  files  through  email 

•  Decisions  where  to  retain  the  documents  under 
control  left  to  producers 
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DVDT  Architecture 


download 


Acc  Plan 
XML  Schema 


1 


V&V  Plan 
XML  Schema 


1 


V&V  Report 
XML  Schema 


1 


Acc  Report 
XML  Schema 


1 


Acc 

Plan 


V&V 

Plan 


V&V 

Report 

4 


Acc 

Report 


Minimal  Set 

Searchable  Data 


►M&S  Catalog 
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DVDT  Screenshots 


{Placeholder  for  tool  demonstration  or  screenshots} 
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My  Projects 


MIL- STD-3  022 

Data  Item  Descriptions 

Policy 

Standards 


TOOL  OPTIONS 


^  My  Profile 
[25  My  Projects 
~l=1  Submit  a  Project 
^  Search 
(D  Log  Out 
(?)  Help 
P1  Feedback 


ft  &  ify  My  Projects  -  DoD  VV&A  Documentation  Tool  (DVDT) 


jf^Page  -  .Tools- 


M&S  Interview 


DoD  W&A  RPG 


After  submitting  VV&A  project  information  a 
list  of  projects  with  versioning  information  is 
provided. 


Name 

Version 

Date  Submitted 

MODSfPJI 

1.3 

8/8/2003  10:45:46  AM 

20 


Project  Information 


DoD  W&A  RPG 


MIL- STD-30  2  2 


Data  Item  Descriptions 


Policy 


Standards 


TOOL  OPTIONS 


gf  My  Profile 


Q)  My  Projects 


Q  Submit  a  Project 


§  Search 


Log  Out 


(?)  Help 


Feedback 


You  must  complete  all  fields  marked  with  a  [R]  in  ALL  TAB  S  before  submitting, 


M &S  Entity  on  Which  WAA  Is  Being  Performed 


rRl  Name:  M0D5IM 


[R]  Version:  1  3 


Description: 


This  year  the  conference  focus  is  on  Decision-Making  in  Complex  Environments 
Speakers  educational  tracks  presentations  and  product  demonstrations  will  center  on 
using  modeling  and  simulation  tools  and  practices  in  decision-making  in  today's  challenging 
operating  environments  attendees  will  learn  about  new  applications  and  practices  and  have 
an  opportunity  to  network  with  other  industry  professionals  One  conference  fee  enables 
exhibitors  and  attendees  to  attend  the  pre-conference  on  Monday  September  15  and  all 
educational  tracks.  An  exciting  cross-cutting  Gaming  Track  is  featured  throughout  each  of 
the  education  tracks  and  will  offer  the  most  up-to-date  developments  in  "Serious  Gaming" 
MGDSIM  World  2008  will  be  held  at  the  Virginia  Beach  Convention  Center  in  Virginia 
Beach  Virginia 


[R]  Intended  Use: 


In  this  workshop  attendees  will  be  introduced  to  using  simulations  and  Stella  to  teach 
Earth  science  as  a  system  Drs  S  Raj  Chaudhury  (CNU)  and  Lin  H  Chambers  (NASA 
Langley)  will  demonstrate  howto  utilize  these  tools  in  a  classroom  system  dynamics 
modeling"  environment  They  will  also  demonstrate  system  dynamics  as  a  computer 
modeling  technique  that  can  be  used  to  produce  dynamic  simulation  models  for  teaching 
and  learning  earth  science  as  a  system  The  basics  of  a  system  dynamics  software 
package  called  Stella  will  be  introduced  and  modeling  exercises  that  utilize  the  Stella 
software  will  be  undertaken  in  the  workshop  No  prior  modeling  experience  or  use  of  Stella 
is  necessary  however  participants  are  required  to  bring  a  lap  top  to  the  workshop 


rirrjflniTflTinn 


fn 


Project  information 
provides  configuration 
management  as  well  as 
discoverable  Metadata 
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IMS  Interview 


Download  Offline  Tool  and/or 

Templates 


DoDW&ARPG 

MIL-STD-3022 

Data  Item  Descriptions 

Policy 

Standards 

My  Profile 

My  Projects 

3  Submit  a  Project 

Search 

(|)  Log  Out 

(?J  Help 

£3  Feedback 

DOWNLOAD  PROJECT  TEMPLATES 

If  you  have  misplaced  the  project  templates  for  T.10DSIM  .  you  can  download  them  again.  Each  of  these  document 
templates  will  only  he  populated  with  data  you  provided  during  your  registration  for  this  specific  project. 


Download  Project  Templates  Only 


DOWNLOAD  W&A  DOCUMENTATION  TOOL 

If  you  need  to  reinstall  the  W&A  Documentation  Tool,  you  can  do  so  by  downloading  the  tool  below  and  following  the 
instructions  provided. 

STEP  1;  DOWNLOAD  W&A  DOCUMENTATION  TOOL 

If  you  need  to  reinstall  the  W&A  Documentation  Tool,  you  can  do  so  by  downloading  the  tool  below  and  following  the 
instructions  provided. 


Download  W&A  Documentation  Tool  and  Project  Templates 


STEP  2:  INSTALL  WS.A  DOCUMENTATION  TOOL 

After  download,  unzip.-  extract  wad  c  etc  cl. zip  to  a  folder  on  your  computer.  Locate  SETUP.EXE  within  the  folder  you 
extracted  to  and  double-click  it  to  run  the  program. 


Upon  install  a  word  template  file  (.dot;  from  the  WA  Documentation  Tool  will  be  copied  to  your  computer  and  if  not 
already  installed.  Visual  Studio  Tools  for  Office  (VSTO)  will  be  added.  The  ;'STO  simply  supports  the  customized  task 
pane  used  amongst  the  documents. 

STEP  3:  START  USING  THE  PROJECT  TEMPLATES 

After  installation,  locate  the  folder  DODMS-WA-2008-3  within  the  area  you  extracted  the  wadoctool.zip  to  and  move  it 
to  an  area  on  your  computer  that  you  will  work  on  the  VVA  project  at  (such  as  My  Documents). 

v 
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XML  via  MS  Word  Interface 


*11  AccreditationPlan.doc  (Read-Only)  Microsoft  Word 


:  File  Edit  View  Insert  Format  Tools  Table  Window  Help 

;j  j  j  j  j.jg  ?a  a  -ji  a  / 


E®® 


J  V  Jjj  iijgj  100%  -  »  4-1  Read  B  liable 


Gridl  »  Arial 


12  t  B  I  * 


Type  a  question  for  help  -r  x 

jE  :E  IfE  IW  'r  ^ 


!  Final  Showing  Markup 


’  Show-'  ^ 


1^. 


Document  Actions 


▼  xl  E 


\  DoD  VV&A  Doc  Tool  il  Website  RPG 


Acc.  Plan  VW  Plan  VW  Report  Acc.  Report 


-  Accreditation  Plan 
i+l  Title  Page 

1  Record  of  Changes 
1  Executive  Summary 
0  Q  1.  Problem  Statement 
D  M  Intended  Use 

immsmm 

)  1.3  M&S  Application 
)  1 .4  Accreditation  Scope 
J1  2.  M&S  Requirements  and  Acceptabilit 
0[)  3.  M8cS  Assumptions,  Capabilities,  Limil 
0j|  4.  Accreditation  Methodology 
F5  5.  Accreditation  Issues 
jl  6.  Key  Participants 
15  7.  Planned  Accreditation  Resources 
Appendix  A.  M&S  Description 
8  Appendix  B.  M&S  Requirements  Trace 
jl  Appendix  C.  Basis  of  Comparison 
|1  Appendix  D.  References 
1  Appendix  E.  Acronyms 
(1  Appendix  F.  Glossary 

>; 


Provides  an  overview  of  the  M&S  for  which  this 
document  is  written  and  discusses  the  level  of 
configuration  control  that  currently  exists  for  the 
M&S. 


AutoShapes  ▼ 


=  a@5 

J  I 


•  £  ...  7 


[enter  table. or  fig y re. . here] 

1.1  Intended  Use 

an 

[enter  table  or  figure  here] 

[] . 

1.2  M&S  Overview 


[enter. t  a b  l.e .  or  fig yre. . here] 

[] 

1.3  M&S  Application 

ID 

[enter  table  or  figure  here] 

0 . 

1.4  Accreditation  Scope 

ID 

[enter,  t  a  b  l.e .  or.  .fig  yre. .  here] 

2  M&S  REQUIREMENTS  AND  ACCEPTABILITY  CRITERIA 

ID 

[enter  t  a  b.I.e .  or.  fig  yre .  here] 


[Print  .0 utli n el .  [Print  Table] .  .(Do ubj e- click . to . se | ect) 


Piioiity  REOe 

M.&  §Re  cjni.r  e  m  e  nt 

c 

[Md],  .[Remove].. 


REQ? 

AC? 

Acceptability  Ciiterion 

\l  1 

[Add]  ..[Remove].. 

AC£  M  M?  Metiic  Measme 


Page  1 


7/15 
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Standardized  Outputs 


’Si  AccreditationPlan.doc  (Read-Only)  -  Microsoft  Word 

BBS 

I  File  Edit  View  Insert  Format  Tools  Table  Window  Help 

Uiudi  J  A  a  /  J  V  ^  11  Q3  -j  IT  100%  -  .  J  Read  Table  Gridl  „  Arial 

-12  t  B  I  ,[1 

oi  help  -  x 

fl  =S  *1?  A.  * 

■  Final  Showing  Markup  »  Show -  ^ 

:  Document  Actions 

noa? 


Arc:  Plan  V8.V  Plan  VW  Report  Acc.  Report 


S  Accreditation  Plan 
f+l  R  Title  Page 

ra  Record  of  Changes 
ra  Executive  Summary 
B  (Hi  1.  Problem  Statement 
P)  1.1  Intended  Use 


)  1.3  M8tS  Application 
_J  1 . 4  Accreditation  S  cope 
U)  2.  M&S  Requirements  and  Acceptabilit 
))  3.  M&S  Assumptions,  Capabilities,  Limit 
)|  4.  Accreditation  Methodology 
5.  Accreditation  Issues 
jg  6.  Key  Participants 
j|  7.  Planned  Accreditation  Resources 
|  Appendix  A.  M&S  Description 
|  Appendix  B.  M&S  Requirements  Trace 
^  Appendix  C.  Basis  of  Comparison 
?)  Appendix  D.  References 
R  Appendix  E.  Acronyms 
|)  Appendix  F.  Glossary 

> 


Provides  an  overview  of  the  M&S  for  which  this 
document  is  written  and  discusses  the  level  of 
configuration  control  that  currently  exists  for  the 
M&S. 


Standards  driven  user 
interface 
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[enter. table,  or  fig .u ire . here] 
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1.  PROBLEM  STATEMENT 


1.1  Intended  Use 


Provides  an  overview  of  the  M&S  for  which  this  document  is  written  and  discusses  the 
level  of  configuration  control  that  currently  exists  for  the  M&S. 


1.2  M&S  Overview 


1.3  M&S  Application 


1.4  A 


2  M&S 

Describe 

resourcin 

likelihooc 

success. 


Priority 

1 


Produces  MS  Word 
documents 

(view  in  print-preview ) 


And  discoverable 
metadata 


□  AccreditationPlan.xml 


1  R<nsl :  Acer  edit  at  ionP  lan  xinlns  :  ns  1  = 11  http  :  /  /metadata,  dod  .mil/mdr/ns/WADocumentation/l .  0" 

Cnsl :  Pro  jectRef  erenceID>D0DMS-WA-2008-3</nsl :  Project ReferenceID> 

Cnsl :  Document  Ref  erenceID>D0DMS-WA-2008-3-9</nsl :  Document  Ref  erenceID> 

4  H  <nsl : AccreditationPlanTit lePage> 

5  H  <nsl : Programldent if icat ion> 

6  r  </nsl : Programldent if icat ion> 

7  R  <nsl : Sponsor ingOrgani zat ionOrPM> 

S  r  </nsl : Sponsor ingOrgani zat ionOrPM> 

9  R  <nsl:HSName>MODSIM 

10  r  </nsl : HSName> 

11  R  <nsl:MSVersion>1.3 

12  -  </nsl : HSVersion> 

13  <nsl : Document Type  userEnteredDocumentType="£ttxA;  " Accreditation  Plan</nsl: 

14  R  <nsl : DocumentTitle>MODSIM  Accreditation  Plan] 
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Consumer  Side  -  M&S  Catalog 


MODIFY  A  PROJECT 


VV&A  Project  information  is  discoverable  as  MSC- 
DMS  compliant  XML.  The  VV&A  XML  schema  is 
an  extension  of  the  DDMS. 


y 


Download 

Templates 


You  must  complete  all  fields  marked  with  a  [R]  in  ALL  TABS  before  submitting. 


M&S  Entity  on  Which  W&A  Is  Being  Performed 


[R]  Name:  MODSIM 


[R]  Version:  1  3 


Search  result  with  Metadata  from  DVDT  Alpha  .01 


Description: 


Simple  Search 


This  year  the  conference  focus  is  on  Decision-Makii_ 


Speakers  educational  tracks  presentations  and  pn  S©3TCfl 
j  using  modeling  and  simulation  tools  and  practices  iif 
operating  environments  attendees  will  learn  about  n| 
an  opportunity  to  network  with  other  industry  professT 
exhibitors  and  attendees  to  attend  the  pre-conferenc| 
educational  tracks.  An  exciting  cross-cutting  GaminI 
the  education  tracks  and  will  offer  the  most  up-to-daf 
(MODSIM  World  2008  will  be  held  at  the  Virginia  Beaf 
Beach  Virginia 


Results  1  -  4  of  4  for  MODS 


[R]  Intended  Use: 


IMQDSIM  Accreditation  Plan 

Title  MODSIM  Accreditation  Plan  ...  Sponsoring  Organization  or  PfvlMODSIM  Project  Reference 
|l  D :  DO  DM  S-WA-20  0  8-3  Document  Reference  ID:DGDMS-WA-2Q08-3-9.  version  1.3 
lxdodwadoctool.crata.ucf  edU'  -2k-  -  cached 


In  this  workshop  attendees  will  be  introduced  to  usil 
Earth  science  as  a  system  Drs  S  Raj  Chaudhury  ( 
Langley)  will  demonstrate  how  to  utilize  these  tools  I 
modeling  environment  They  will  also  demonstrate  j 
modeling  technique  that  can  be  used  to  produce  dyrl 
and  learning  earth  science  as  a  system  The  basics! 
package  called  Stella  will  be  introduced  and  modeling 
software  will  be  undertaken  in  the  workshop.  No  prio| 
is  necessary  however  participants  are  required  to  t 


Title  MODSIM  Accreditation  Plan 

Description:  In  this  workshop,  attendees  will  he  introduced  to  using  simulations  and  Stella  to  teach  Earth  science 
IChaudhury  (GNU)  and  Lin  H.  Chambers  fNASA  Langley)  will  demonstrate  howto  utilize  these  tools  in  a  classroom,  i 
|modelingIiVa  environment.  They  will  also  demonstrate 
Version:  1  3 


* 


□esdTption 


KeyW&fds 


IN  aval  Mine  Warfare  Simulation 

...  In  put:  Systems  performance  characteristics  environmental  data  and  operation 
Jplans  and  scenarios.  Source  Code. MODSIM  III  Visual  Basic.  C.  ... 
nso-navy  mikmdex.cfm%3Frid%3DMNS N 101QQG4  -  5k  -  -  cacjied 


Rights 


Title  Naval  Mine  Warfare  Simulation 
Description  NMWS  was  developed  by  PEO  (MlW)to  support  MS  II  Analysis  of  Alternatives  in  the  acquisition  proc 
Jevent -driven  simulation  was  developed  using  the  Shlaer/Mellor  object-oriented  methodology 
Sponsor:  MAVSEASYSCOM 
Acronym:  NMWS 

Releaseability  This  resource  is  UNCLASSIFIED 


Beta  Testing 


•  Alpha  testing  (internal)  conducted  29  Sep-3  Oct  2008 

•  Beta  testing  (external)  scheduled  to  start  3-7  Nov  2008 


VV6A0FM65 


NAV5EA  QAHLGREN  ACCREDITATION  TEAM 


Why  is  VV&A  Information  Important? 


•  VV&A  information  tells  consumers  about 

-  M&S  assumptions  (simplifications  and  potential  failure  points) 

-  M&S  capabilities  (what  the  M&S  can  be  used  to  do) 

-  M&S  limitations  (what  it  should  not  be  used  to  do) 

•  Consistently  documenting  VV&A  information  across  DoD  yields 
many  returns 

-  Discoverable  VV&A  information 
saves  time  and  money  finding  an 
M&S  to  satisfy  a  need 

-  VV&A  documents  provide 
evidence  to  determine  credibility 
of  M&S  results  to  support  an 
intended  use 

-  Credible  M&S  results  can  be 
defended  and  used  with 
confidence 


CREDIBILITY 

V 

V 
& 


\  o  /  \ 

RELEVANCE  °*  CONFIDENCE 

A 


M&S 
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Discoverable  VV&A  Document  Metadata 


s  s 

■  ■  ■  Producer 

Consumer 

Consumer 

V  J 

discovers  VV&A  document  information 

DVDT 


M&S 

resources  in 
the  GIG 
Enterprise 

M&S 

Resources 
described  by 
metadata 
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M&S  Search  Tool 


Need  W&A  information? 


Consumer 


search 


W&A 

information 


Using  the  power  of  search  and  discovery  capability 
to  conduct  focused  federated  searches  for 
information  about  W&A  documents 
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Summary 


•  Updated  the  Systems  Engineering  Community 
on  the  DoD  M&S  Project,  Standardized 
Documentation  for  VV&A 

•  Provided  information  about  related  policy, 
guidance,  and  standards 

•  Discussed  Discovery  Metadata  and  discovery 
mechanisms 

•  Described  and  demonstrated  DVDT 


Using  the  DVDT  to  document  implementation  of 
VV&A  processes  enables  systems  engineers  to 
use  M&S  results  with  confidence 
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Point  of  Contact 


Kevin  Charlow 


Space  and  Naval  Warfare  Systems  Center 

Atlantic 

P.O.  Box  190022 
North  Charleston,  SC  29419 
843-218-5372 
kevin.charlow@navy.mil 
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•  Charlow,  K.,  Broyles,  D.,  Blais,  C.,  and  Stutzman,  M.:  “Standardized  Documentation  For 
Verification,  Validation,  And  Accreditation  (VV&A)  —  Helping  Assure  Mission  Success,”  Paper 
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Acronyms 


CRP 

Comment  Resolution  Panel 

DDMS 

DoD  Discovery  Metadata  Specification 

DG 

Drafting  Group 

DID 

Date  Item  Description 

DMSP 

DoD  M&S  Project 

DoD 

Department  of  Defense 

DVDT 

DoD  VV&A  Documentation  Tool 

GIG 

Global  Information  Grid 

M&S 

Modeling  and  Simulation,  model(s)  and  simulation(s) 

M&S  CO 

M&S  Coordination  Office 

M&S  IPT 

M&S  Integrated  Product  Team 

M&S  SC 

M&S  Steering  Committee 

MIL-STD 

Military  Standard 

MORS 

Military  Operational  Research  Society 

MSC  DMS 

M&S  Community  of  Interest  Discovery  Metadata  Specification 

USD(AT&L) 

Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics 

VV&A 

Verification,  Validation,  and  Accreditation 

WG 

Working  Group 

XML 

Extensible  Markup  Language 
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Overview 


■  Purpose 

■  TECHSOFT 

■  Standards-based  Process  Improvement  Success 

■  Why  Harmonize? 

□  Issues 
Impacts  to  you 

■  SE/SW  LCP  Alignment  and  Integration 

□  Path 

□  Concepts 

Where  we  are  today 

How  we  got  here  -  Key  changes  in  15288  &  12207 

■  Large  Scale  Harmonization 

■  Benefits  Summary 
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Technical  Software  Services,  Inc. 


Show  how  the  key  changes  in  the  alignment  of  a 
foundational  systems/software  standards  set 
(ISO/IEC/IEEE  15288  and  ISO/IEC/IEEE  12207)  facilitates 
integrated  systems  and  software  engineering,  project 
management,  and  acquisition 


SEIParcnor 
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Technical  Software  Services,  Inc. 
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Who  We  Are 

■  Founded  in  1990 

■  Based  in  Pensacola,  Florida 

Presence  in  Charleston,  SC 

■  Primarily,  a  DoD  Contractor 

■  Experienced  Staff 

High  %  Masters  level  personnel 

□  Majority  with  Security 
Clearances 

SEI-Authorized  CMMI®  Lead 
Appraisers 

SEI-Authorized  CMMI® 
Instructors 

International  SE/SW  Standards 
Expertise 
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fECHSOFT 


What  We  Do 

Systems  &  Software 
Development 

Database  Applications 

Security  /  IA 

Web  Development 

Network  Engineering/Hosting 

Training 

Process  Engineering/Process 
Improvement 

□  CMMI® 

□  SEI  Partner 
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Standards-based  Process  Improvement 

Example  of  a  Successful  Approach 


Source:  N65236-ENGOPS-BRIEF-0068-1.1,  Standardization  of  Systems  Engineering  &  Project  Management  Using  CMMI,  M.T.  Kutch,  Jr.,  17JUL08 
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Full  OPD,  But  Today’s  Focus:  15288/1 2207 


This  SSC  has 
15288  and 
12207-based 
SE/SWE 
Technical 
Processes 


SPHMl 

Y 


SSC-C  SE  Revitalization  Plan 
Aligned  with  DoD  SE  Revitalization 


Systems  Center 
Charleston 


Elements  of  SSC-C  SE  Revitalization 


Policy  /  Guidance 

■ 

Training  /  Education 

■ 

Assessment  &  Support 

L 


Intro  to  PI  WBT 


-L 


CMMI®  Level  2 


SE  101  WBT 


CMMI®  Level  3 


SE  Fundamentals 


CMMI®  Level  4/5 


SSU-C  JUU-IUIellnt 
Process  Manual 


SE  for  Managers 


Project  Reviews 


EPO  Website 


Project  &  Process 
Workshop 


Balanced  Scorecard 


ePIan  Builder 


Intro  to  Software  Engr. 


Lean  Six  Sigma 


DO  Underway 

I  |  Completed/Ongoing 


Architecture  Dev.  WBT 


Integrated  Product 
Teams 


— 

Certification/Degrees 


IT  Tools 


IHCHSOFT 


With  Extensive 
OPA  Support 


Source:  N65236-ENGOPS-BRIEF-0048-1.2,  Tools  and  Resources  to  Enable 
Systems  Engineering  Improvement,  M.T.  Kutch,  Jr.  &  M.  Knox,  NOV07 
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Process  Asset  Library 


Standard 

Processes 


*  Process 
Manuals 

*  SOPs 

*  Sample 
Documents 

*  Templates 


Project  Life 
Cycles 


•  Waterfall 

•  Spiral 

•  incremental 


Library  ol 
Process-Related 
Documentation 


•  Sample 
Documents 

•Searchable 

Database 


Tailoring 

Criteria 


•  Policy 

•  Sample 
Documents 


*  Project  Data 

•  Corporate  (Data 
•Cost 


Techsoft 


So  what’s  the  problem  with  15288  and  12207 


ISO/IEC  15288:2002  ISO/IEC  12207:1995 


Enterprise  Processes 

Enterprise  Environment 
Management  Process 


Investment  Management 
Process 


System  Life  Cycle  Processes 
Management  Process 


Resource  Management 
Process 


Quality  Management 
Process 


Agreement  Processes 

Acquisition  Process 


Supply  Process 


Project 

Processes 

Project  Planning 
Process 


Project  Assessment 
Process 


Project  Control 
Process 


Decision-making 

Process 


Risk  Management 
Process 


Configuration 
Management  Process 


Information 
Management  Process 


Technical  Processes 

Stakeholder  Requirements 
Definition  Process 

Requirements  Analysis 
Process 

Architectural  Design 
Process 


Implementation  Process 


Integration  Process 


Verification  Process 


Transition  Process 


Validation  Process 


Operation  Process 


Maintenance  Process 


Disposal  Process 


5.  PRIMARY 

LIFE  CYCLE  PROCESSES 


5.1  Acquisition 


5.2  Supply 


6.  SUPPORTING 
LIFE  CYCLE  PROCESSES 


6.1  Documentation 


6.2  Configuration 
Management 


6.3  Quality 
Assurance 


6.4  Verification 


6.5  Validation 


6.6  Joint  Review 


6.7  Audit 


6.8  Problem  Resolution 


Using  Them  Together! 

•  Conflicting  terms  and  definitions 

•  Overlapping,  yet  distinct  processes 

•  Different  process  architectures 

•  Different  levels  of  prescription 


7.  ORGANIZATIONAL  LIFE  CYCLE  PROCESSES 


7.1  Management 


7.2  Infrastructure 


7.3  Improvement 


7.4  Training 


SEIftarcncr 

IIIechsoft 

Technics!  Software  Services,  Ire. 


Unintegrated  12207  amendments 
from  2002  and  2004  are  difficult  to 
use  and  also  not  adopted  by  IEEE 
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Why  You  Should  Care 


■  Leverage  the  Commonalties 

Identify  and  explain  the  differences 
Use  the  interfaces 

■  Promote  Communication  and  Team  Integration 

Identify  strengths,  views,  and  appropriate  focused  implementations 

□  Reduce  us/them,  finger-pointing,  stove-piping 

■  Improve  Resource  Performance 

□  Personnel,  Processes,  Tools,  Services 

■  Lower  Costs 

Reduce  redundancy  and  inefficiency 


Benefits  of  Standards  Harmonization 

Supports  Integration,  Facilitates  Management,  Simplifies  Acquisition 


SEIParcncr 


Technical  Software  Services,  Inc, 
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15288-12207  Harmonization  Path 


Harmonization  Study  Group 

Ikpon  to  the  Advisory  tinni|>  Mipiii^j] 


Doug  Thiele 
WG  7  Convert 


1 

’n^-’ns 


Studies 


Harmonization 

ISO/I  EC  15288  &  ISO/I  EC  12207  Revisions 


STOCKHOLM  meeting 


Project  Editor  report 


Implementation  hits  a  snag 
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“Harmonization  Lite” 


— A  Proposal 


James  W.  Hoorn,  C5DP 
The  MITRE  Corporation 


IEEE  Computer  Society  Liaison  to  tSOrlEC  JTC  1/SC  7 
Ji  Moore jSrraoo.org 

do  not 


Eat  that 
elephant 
one  bite 
at  a  time! 


ISO/IEC  JTC  1/SC  7/WG  7  N0868 
2005-05-27 


Harmonization  revised  concept 


Alignment 


Harmonization 

Publicity 


Integration 


Life  cycle  concepts 
V  Process  Architecture  J 
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Concept  for  the  Harmonized  Set 


► 


2005 


Source:  ISO/IEC  JTC1/SC7  WG7  N01025  Briefing  Material,  24MAY07 
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Where  We  Are  Today 


Nearly  identical  process  models 


System 

Level 

Processes 

INTERNATIONAL  ISO/IEC 

STANDARD  15288 

IEEE 

Std  15288-2008 

Second  eciticn 
2QQ3-D2-D1 

INTERNATIONAL  ISO/IEC 

STANDARD  12207 

IEEE 

Std  12207-2003 

Second  eeiticn 
2CK33-D2-D1 

System 
Processes 
Specialized 
To  Software 
and 

Software- 

Specific 

Systems  and  software  engineering  — 
System  life  cycle  processes 

Systems  and  software  engineering  —  j 

Software  life  cycle  processes 

fngenrene  ties  systemes  ei  ou  icgiciel  —  Processus  du  cycle  de  vie 
du  systeme 

inger.ierie  des  systemes  ei  du  iogicie}  —  Processus  du  cycle  de  vie 
du  tagidel 

Processes 

Life  Cycle  Concepts 
Process  Concepts 
LC  Models,  Stages 


DRAFT 


ISO/I  EC  JTC  1/SC  7 

Date:  2008-08-11 

ISO/IEC  DTR  24748-1 

ISO/IEC  JTC  1/SC  7/WG  7  Nil 40 
Secretariat:  SCC 


Systems  and  software  engineering - Guide  for  life  cycle 

management 

Ingenierle  systemes  et  logiciel - Guide  pour  gestion  du  cycle  de  vie 

It  is  the  intention  of  this  project  to  create  a  Technical  Report  of  Type  3  that  may  be  made  freely  available  in  accordance 
with  the  provisions  of  JTC  1  N  7269  and  Sendai  Resolution  32.  In  particular,  the  document  has  the  following 


LC  Adaptation 
Domains,  Disciplines, 
&  Specialties 
Prior  Version  Transition 


SEIParcncr 
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Technical  Software  Services,  Ire. 
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Relations  of  Process  Constructs  among  ISO/IEC  12207:1995 
and  its  Amendments,  15288:2002, 15288:2008  &  12207:2008 


Source:  Anatol  Kark,  National  Research  Council,  Canada 


12207/15288:2008  Process  Constructs 


Processes  require  a  purpose  and  outcome.  All 
processes  have  at  least  one  activity.  The  processes, 
with  their  statements  of  purpose  and  outcomes, 
constitute  a  Process  Reference  Model  (PRM). 


Activities  are  constructs  for  grouping  together 
related  tasks.  The  activities  provide  a  means  to  look 
at  related  tasks  within  the  process  to  improve 
understanding  and  communication  of  the  process.  If 
an  activity  is  cohesive  enough,  it  can  be  converted 
to  a  (lower  level)  process  by  defining  a  purpose  and 
a  set  of  outcomes. 


A  task  is  a  detailed  provision  for  implementation  of  a 
process.  It  may  be  a  requirement  (“shall”),  a 
recommendation  (“should”),  or  a  permission  (“may”). 


Notes  are  used  when  there  is  a  need  for  explanatory 
information  to  better  describe  the  intent  or 
mechanics  of  a  process.  Notes  provide  insight 
regarding  potential  implementation  or  areas  of 
applicability  such  as  lists,  examples  and  other 
considerations. 


Adapted  from  ISO/IEC  JTC1/SC7  WG7  N1025  briefing  material 


Normative 


J 

1  Informative 
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The  Life  Cycle  Processes  of  15288:2002 


Agreement 


Acquisition  Process 

Supply  Process 

Enterprise 

Enterprise  Environment 
Management  Process 


Investment  Management 
Process 


System  LC  Processes 
Management  Process 


Resource  Management 
Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment 
Process 


Project  Control  Process 


Decision-Making  Process 


Risk  Management 
Process 


Configuration 
Management  Process 


Information  Management 
Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 


Requirements  Analysis 
Process 


Architectural  Design 
Process 


Implementation  Process 


Integration  Process 


Verification  Process 


Transition  Process 


Validation  Process 


Operation  Process 


Maintenance  Process 


Disposal  Process 


Source:  WG7  N1111;  Adapted  by  Jim  Moore,  MITRE  Corporation  from  chart  by  Anatol  Kark,  National  Research  Council,  Canada 
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Building  15288:2008  -  Activities  and  Tasks 

■\ 


Activity-Task  allocation  is  new  to  15288:2008 
Provides  structural  alignment  with  12207 


Agreement 


Acquisition  Process 


Supply  Process 


Enterprise 


Enterprise  Environment 
Management  Process 


Investment  Management 
Process 


System  LC  Processes 
Management  Process 


Resource  Management 
Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment 
Process 


Project  Control  Process 


Decision-Making  Process 


Risk  Management 
Process 


Configuration 
Management  Process 


Information  Management 
Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 


Requirements  Analysis 
Process 


Architectural  Design 
Process 


Implementation  Process 


Integration  Process 


Verification  Process 


Transition  Process 


Validation  Process 


Operation  Process 


Maintenance  Process 


Disposal  Process 


0..* 

Process 

Name,  Purpose, 
Outcome(s) 

ri 

A  * 

h 

Activity 


Name 


1 

1..* 


Task 


> 


Normative 


J 


Note 

J 

Informative 


Adapted  from  WG7  Nil  11;  Source:  Jim  Moore,  MITRE  Corporation  and  Anatol  Kark,  National  Research  Council,  Canada 
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Building  15288:2008  -  Technical  Processes 


15288:2008  has  the  same  set  of  technical  processes  as  15288:2002 


Agreement 


Acquisition  Process 

Supply  Process 

Enterprise 

Enterprise  Environment 
Management  Process 


Investment  Management 
Process 


System  LC  Processes 
Management  Process 


Resource  Management 
Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment 
Process 


Project  Control  Process 


Decision-Making  Process 


Risk  Management 
Process 


Configuration 
Management  Process 


Information  Management 
Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 


Requirements  Analysis 
Process 


Architectural  Design 
Process 


Implementation  Process 


Integration  Process 


Verification  Process 


Transition  Process 


Validation  Process 


Operation  Process 


Maintenance  Process 


Disposal  Process 


Technical 

Stakeholder  Reqmts 
Definition  Process 

Requirements  Analysis 
Process 

Architectural  Design 
Process 

Implementation  Process 

Integration  Process 

Verification  Process 

Transition  Process 

Validation  Process 

Operation  Process 

Maintenance  Process 

Disposal  Process 

Source:  WG7  N1111;  Adapted  by  Jim  Moore,  MITRE  Corporation  from  chart  by  Anatol  Kark,  National  Research  Council,  Canada 
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Building  15288:2008  -  Project  Processes 


15288:2008  has  a  similar  set  of  project  processes  as  15288:2002 


Agreement 


Acquisition  Process 


Supply  Process 


Enterprise 

Enterprise  Environment 
Management  Process 


Investment  Management 
Process 


System  LC  Processes 
Management  Process 


Resource  Management 
Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management 
Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 


Requirements  Analysis 
Process 


Architectural  Design 
Process 


Implementation  Process 


Integration  Process 


Verification  Process 


Transition  Process 


Validation  Process 


Operation  Process 


Maintenance  Process 


Disposal  Process 


Source:  WG7  N1111;  Adapted  by  Jim  Moore,  MITRE  Corporation  from  chart  by  Anatol  Kark,  National  Research  Council,  Canada 
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Building  15288:2008  -  Project-Enabling  Processes 

15288:2008  has  a  similar  set  of  project-enabling  processes  as  15288:2002 


Agreement 


Acquisition  Process 


Supply  Process 


Organizational 

Project-Enabling 


Life  Cycle  Model 
Management  Process 


Infrastructure 
Management  Process 


Project  Portfolio 
Management  Process 


Human  Resource 
Management  Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management 
Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 


Requirements  Analysis 
Process 


Architectural  Design 
Process 


Implementation  Process 


Integration  Process 


Verification  Process 


Transition  Process 


Validation  Process 


Operation  Process 


Maintenance  Process 


Disposal  Process 


Source:  WG7  N1111;  Adapted  by  Jim  Moore,  MITRE  Corporation  from  chart  by  Anatol  Kark,  National  Research  Council,  Canada 
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Building  15288:2008  -  Agreement  Processes 


15288:2008  has  the  same  set  of  agreement  processes  as  15288:2002 


Agreement 


Acquisition  Process 


Supply  Process 


Agreement 


Acquisition  Process 


Supply  Process 


Organizational 

Project-Enabling 


Life  Cycle  Model 
Management  Process 


Infrastructure 
Management  Process 


Project  Portfolio 
Management  Process 


Human  Resource 
Management  Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management 
Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 


Requirements  Analysis 
Process 


Architectural  Design 
Process 


Implementation  Process 


Integration  Process 


Verification  Process 


Transition  Process 


Validation  Process 


Operation  Process 


Maintenance  Process 


Disposal  Process 


Source:  WG7  N1111;  Adapted  by  Jim  Moore,  MITRE  Corporation  from  chart  by  Anatol  Kark,  National  Research  Council,  Canada 
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The  Life  Cycle  Processes  of  15288:2008 


Agreement 


Acquisition  Process 


Supply  Process 


Organizational 

Project-Enabling 

Life  Cycle  Model 
Management  Process 


Infrastructure 
Management  Process 


Project  Portfolio 
Management  Process 


Human  Resource 
Management  Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management 
Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 


Requirements  Analysis 
Process 


Architectural  Design 
Process 


Implementation  Process 


Integration  Process 


Verification  Process 


Transition  Process 


Validation  Process 


Operation  Process 


Maintenance  Process 


Disposal  Process 


Source:  WG7  Nil  11;  Adapted  by  Jim  Moore,  MITRE  Corporation  from  chart  by  Anatol  Kark,  National  Research  Council,  Canada 
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The  Life  Cycle  Processes  of  12207:1995 


5.  PRIMARY 

LIFE  CYCLE  PROCESSES 


5.1  Acquisition 


5.2  Supply 


6.  SUPPORTING 
LIFE  CYCLE  PROCESSES 


6.1  Documentation 


6.2  Configuration 
Management 


6.3  Quality 
Assurance 


6.4  Verification 


6.5  Validation 


6.6  Joint  Review 


6.7  Audit 


6.8  Problem  Resolution 


7.  ORGANIZATIONAL  LIFE  CYCLE  PROCESSES 


7.1  Management 


7.2  Infrastructure 


7.3  Improvement 


7.4  Training 


The  Familiar  1995 
LCP  Categories 
Process  Structure 
and  Titles 


Adapted  from  WG7  Nil  11  briefing  material 
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The  Life  Cycle  Processes  of  12207:1995 


Primary 


Acquisition  Process 


Supply  Process 


Organizational 


Improvement  Process 


Management  Process 


Infrastructure  Process 


Training  Process 


Primarily 

organization- 

oriented 


Primarily 

project- 

oriented 


System  Integration 


System  Qualification 
Testing 


Software  Installation 


Software  Acceptance 
Support 


Operation  Process 


Maintenance  Process 


Development  Process 


!  System  Requirements  \ 
\  Analysis  \ 

i _ j 

i - 1 

!  System  Architectural  \ 
\  Design 


Process  Implementation 


Software  Requirements 
Analysis 


Software  Architectural 
Design 


Software  Detailed  Design 


Software  Coding  & 
Testing 


Software  Integration 


Software  Qualification 
Testing 


Supporting 


Documentation 
Management  Process 


Configuration 
Management  Process 


Quality  Assurance 
Process 


Verification  Process 


Validation  Process 


Joint  Review  Process 


Audit  Process 


Problem  Resolution 
Process 
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Box  with  dashed  border 
was  an  Activity  in  1995 


SBftrtncr 


HIechsoft 

Technical  Software  Services,  Inc. 


12207  Amd. 1:2002  and  Amd. 2:2004 


■  Defined  a  Process  Reference  Model  (PRM)  for  12207 

□  Process  Name,  Purpose,  and  Outcomes 

■  Restructured  processes  to  provide  higher  granularity 

Introduced  sub-processes  (e.g  based  on  Development  activities) 

Improvement,  Human  Resource,  Acquisition,  Supply,  Development, 
Operation,  Management 

■  Introduced  extensions,  elaborations  and  new  processes 

e.g.  to  better  support  process  assessment  (15504-2), 
usability(1 3407),  measurement  (15939),  product  evaluation(14598), 
and  reuse/asset  management  (IEEE  1517) 

■  Added  activities  and  tasks  for  8  new  processes 

■  Made  some  corrections 


Generally  aligned  and  incorporated  in  body  of  revised  12207 
Several  sub-processes  allocated  as  lower-level  PRM  only  processes 


SEIParcncr 


Technical  Software  Services,  Ire. 
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The  Life  Cycle  Processes  of  12207:2008 


Agreement 


Acquisition  Process 


Supply  Process 


Organizational 

Project-Enabling 

Life  Cycle  Model 
Management  Process 


Infrastructure 
Management  Process 


Project  Portfolio 
Management  Process 


Human  Resource 
Management  Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management 
Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 


System  Requirements 
Analysis  Process 


System  Architectural 
Design  Process 


Implementation  Process 


System  Integration 
Process 


System  Qualification 
Testing  Process 


Software  Installation 
Process 


Software  Acceptance 
Support  Process 


Software  Operation 
Process 


Software  Maintenance 
Process 


Software  Disposal 
Process 


< 


SW  Implementation 


Software  Implementation 
Process 

Software  Requirements 
Analysis  Process 

Software  Architectural 
Design  Process 

Software  Detailed 
Design  Process 

Software  Construction 
Process 

Software  Integration 
Process 

Software  Qualification 
Testing  Process 

SW  Reuse 


Domain  Engineering 
Process 


Reuse  Asset 
Management  Process 


Reuse  Program 
Management  Process 
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SW  Support 

Software  Documentation 
Management  Process 


Software  Configuration 
Management  Process 


Software  Quality 
Assurance  Process 


Software  Verification 
Process 


Software  Validation 
Process 


Software  Review  Process 


Software  Audit  Process 


Software  Problem 
Resolution  Process 


sh  _  SEJHarcncr 

IIIechsoft 

Technical  Software  Services,  Inc. 


Building  12207:2008  -  System  Context 


Structural  alignment  with  15288  system  level  categories 


Agreement 


Acquisition  Process 


Supply  Process 


Organizational 

Project-Enabling 

Life  Cycle  Model 
Management  Process 


Infrastructure 
Management  Process 


Project  Portfolio 
Management  Process 


Human  Resource 
Management  Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management 
Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 


System  Requirements 
Analysis  Process 


System  Architectural 
Design  Process 


Implementation  Process 


System  Integration 
Process 


System  Qualification 
Testing  Process 


Software  Installation 
Process 


Software  Acceptance 
Support  Process 


Software  Operation 
Process 


Software  Maintenance 
Process 


Software  Disposal 
Process 


< 


SW  Implementation 


Software  Implementation 
Process 

Software  Requirements 
Analysis  Process 

Software  Architectural 
Design  Process 

Software  Detailed 
Design  Process 

Software  Construction 
Process 

Software  Integration 
Process 

Software  Qualification 
Testing  Process 

SW  Reuse 


Domain  Engineering 
Process 


Reuse  Asset 
Management  Process 


Reuse  Program 
Management  Process 
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SW  Support 

Software  Documentation 
Management  Process 


Software  Configuration 
Management  Process 


Software  Quality 
Assurance  Process 


Software  Verification 
Process 


Software  Validation 
Process 


Software  Review  Process 


Software  Audit  Process 


Software  Problem 
Resolution  Process 


sh  _  SEJHarcncr 
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Building  12207:2008  -  System  Context 


System  Context  Processes  based  on  15288  Processes 


Agreement 


Acquisition  Process 


Supply  Process 


Organizational 

Project-Enabling 

Life  Cycle  Model 
Management  Process 


Infrastructure 
Management  Process 


Project  Portfolio 
Management  Process 


Human  Resource 
Management  Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management 
Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 


System  Requirements 
Analysis  Process 


System  Architectural 
Design  Process 


Implementation  Process 


System  Integration 
Process 


System  Qualification 
Testing  Process 


Software  Installation 
Process 


Software  Acceptance 
Support  Process 


Software  Operation 
Process 


Software  Maintenance 
Process 


Software  Disposal 
Process 


< 


SW  Implementation 


Software  Implementation 
Process 

Software  Requirements 
Analysis  Process 

Software  Architectural 
Design  Process 

Software  Detailed 
Design  Process 

Software  Construction 
Process 

Software  Integration 
Process 

Software  Qualification 
Testing  Process 

SW  Reuse 


Domain  Engineering 
Process 


Reuse  Asset 
Management  Process 


Reuse  Program 
Management  Process 
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SW  Support 

Software  Documentation 
Management  Process 


Software  Configuration 
Management  Process 


Software  Quality 
Assurance  Process 


Software  Verification 
Process 


Software  Validation 
Process 


Software  Review  Process 


Software  Audit  Process 


Software  Problem 
Resolution  Process 


sh  _  SEJHarcncr 
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Building  12207:2008  -  System  Context 

Include  12207  Organizational  Processes:  Improvement,  Infrastructure, 


Human  Resource/Training,  Management 


Agreement 


Acquisition  Process 


Supply  Process 


Organizational 

Project-Enabling 


Life  Cycle  Model 
Management  Process 


Infrastructure 
Management  Process 


Project  Portfolio 
Management  Process 


Human  Resource 
Management  Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management 
Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 


System  Requirements 
Analysis  Process 


System  Architectural 
Design  Process 


Implementation  Process 


System  Integration 
Process 


System  Qualification 
Testing  Process 


Software  Installation 
Process 


Software  Acceptance 
Support  Process 


Software  Operation 
Process 


Software  Maintenance 
Process 


Software  Disposal 
Process 


< 


SW  Implementation 


Software  Implementation 
Process 

Software  Requirements 
Analysis  Process 

Software  Architectural 
Design  Process 

Software  Detailed 
Design  Process 

Software  Construction 
Process 

Software  Integration 
Process 

Software  Qualification 
Testing  Process 

SW  Reuse 


Domain  Engineering 
Process 


Reuse  Asset 
Management  Process 


Reuse  Program 
Management  Process 


SW  Support 

Software  Documentation 
Management  Process 


Software  Configuration 
Management  Process 


Software  Quality 
Assurance  Process 


Software  Verification 
Process 


Software  Validation 
Process 


Software  Review  Process 


Software  Audit  Process 


Software  Problem 
Resolution  Process 


Adapted  from  WG7  N1111; 

Adapted  15288  Outcome/s 

One  or  more 

Blended  12207  &  15288 

One  or  more 

12207-based  Outcome/s 
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Activities,  Tasks 

12207  Outcomes 

Activities  and  Tasks 

15288  Outcomes 

Activities,  Tasks 

Building  12207:2008  -  System  Context 


Risk  Management  from  16085  and  Measurement  from  15939  are  added 


Technical 


Stakeholder  Reqmts 
Definition  Process 


System  Requirements 
Analysis  Process 


System  Architectural 
Design  Process 


Implementation  Process 


System  Integration 
Process 


System  Qualification 
Testing  Process 


Software  Installation 
Process 


Software  Acceptance 
Support  Process 


Software  Operation 
Process 


Software  Maintenance 
Process 


Software  Disposal 
Process 


Agreement 


Acquisition  Process 


Supply  Process 


Organizational 

Project-Enabling 

Life  Cycle  Model 
Management  Process 


Infrastructure 
Management  Process 


Project  Portfolio 
Management  Process 


Human  Resource 
Management  Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management 
Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


SW  Implementation 


Software  Implementation 
Process 


Software  Requirements 
Analysis  Process 


Software  Architectural 
Design  Process 


Software  Detailed 
Design  Process 


Software  Construction 
Process 


Software  Integration 
Process 


Software  Qualification 
Testing  Process 


SW  Reuse 


Domain  Engineering 
Process 


Reuse  Asset 
Management  Process 


Reuse  Program 
Management  Process 


SW  Support 


Software  Documentation 
Management  Process 


Software  Configuration 
Management  Process 


Software  Quality 
Assurance  Process 


Software  Verification 
Process 


Software  Validation 
Process 


Software  Review  Process 


Software  Audit  Process 


Software  Problem 
Resolution  Process 
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One  or  more 

Blended  12207  &  15288 

One  or  more 

12207-based  Outcome/s 
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Activities,  Tasks 

12207  Outcomes 

Activities  and  Tasks 

15288  Outcomes 

Activities,  Tasks 

Building  12207:2008  -  System  Context 

Risk  Management  and  Measurement  are  now  almost  identical  to  15288 
12207  Acquisition  and  Supply  are  blended  with  15288  Agreement  Processes 


Agreement 


Acquisition  Process 


Supply  Process 


Organizational 

Project-Enabling 


Life  Cycle  Model 
Management  Process 


Infrastructure 
Management  Process 


Project  Portfolio 
Management  Process 


Human  Resource 
Management  Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management 
Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 


System  Requirements 
Analysis  Process 


System  Architectural 
Design  Process 


Implementation  Process 


System  Integration 
Process 


System  Qualification 
Testing  Process 


Software  Installation 
Process 


Software  Acceptance 
Support  Process 


Software  Operation 
Process 


Software  Maintenance 
Process 


Software  Disposal 
Process 
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SW  Implementation 


Software  Implementation 
Process 


Software  Requirements 
Analysis  Process 


Software  Architectural 
Design  Process 


Software  Detailed 
Design  Process 


Software  Construction 
Process 


Software  Integration 
Process 


Software  Qualification 
Testing  Process 


SW  Reuse 


Domain  Engineering 
Process 


Reuse  Asset 
Management  Process 


Reuse  Program 
Management  Process 


SW  Support 


Software  Documentation 
Management  Process 


Software  Configuration 
Management  Process 


Software  Quality 
Assurance  Process 


Software  Verification 
Process 


Software  Validation 
Process 


Software  Review  Process 


Software  Audit  Process 


Software  Problem 
Resolution  Process 
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One  or  more 

12207-based  Outcome/s 
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Activities,  Tasks 

12207  Outcomes 

Activities  and  Tasks 

15288  Outcomes 

Activities,  Tasks 

Building  12207:2008  -  System  and  Software 


Development  Activities  form  System  Context  and  Software  Specific  Processes 


Technical 


Stakeholder  Reqmts 
Definition  Process 


System  Requirements 
Analysis  Process 


System  Architectural 
Design  Process 


Implementation  Process 


System  Integration 
Process 


System  Qualification 
Testing  Process 


Software  Installation 
Process 


Software  Acceptance 
Support  Process 


Software  Operation 
Process 


Software  Maintenance 
Process 


Software  Disposal 
Process 


Agreement 


Acquisition  Process 


Supply  Process 


Organizational 

Project-Enabling 

Life  Cycle  Model 
Management  Process 


Infrastructure 
Management  Process 


Project  Portfolio 
Management  Process 


Human  Resource 
Management  Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management 
Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


SW  Implementation 


Software  Implementation 
Process 

Software  Requirements 
Analysis  Process 

Software  Architectural 
Design  Process 

Software  Detailed 
Design  Process  1 

Software  Construction 
Process 

Software  Integration 
Process 

Software  Qualification 
Testing  Process 

SW  Reuse 


Domain  Engineering 
Process 


Reuse  Asset 
Management  Process 


Reuse  Program 
Management  Process 


SW  Support 

Software  Documentation 
Management  Process 


Software  Configuration 
Management  Process 


Software  Quality 
Assurance  Process 


Software  Verification 
Process 


Software  Validation 
Process 


Software  Review  Process 


Software  Audit  Process 


Software  Problem 
Resolution  Process 
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12207-based  Outcome/s 
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Activities,  Tasks 

12207  Outcomes 

Activities  and  Tasks 

15288  Outcomes 

Activities,  Tasks 

Building  12207:2008  -  System  Context 


12207  Operation  and  Maintenance  Processes  complete  the  System  Context 


Technical 


Stakeholder  Reqmts 
Definition  Process 


System  Requirements 
Analysis  Process 


System  Architectural 
Design  Process 


Implementation  Process 


System  Integration 
Process 


System  Qualification 
Testing  Process 


Software  Installation 
Process 


Software  Acceptance 
Support  Process 


Software  Operation 
Process 


Software  Maintenance 
Process 


Software  Disposal 
Process 


Agreement 


Acquisition  Process 


Supply  Process 


Organizational 

Project-Enabling 

Life  Cycle  Model 
Management  Process 


Infrastructure 
Management  Process 


Project  Portfolio 
Management  Process 


Human  Resource 
Management  Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management 
Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


SW  Implementation 


Software  Implementation 
Process 

Software  Requirements 
Analysis  Process 

Software  Architectural 
Design  Process 

Software  Detailed 
Design  Process  1 

Software  Construction 
Process 

Software  Integration 
Process 

Software  Qualification 
Testing  Process 

SW  Reuse 


Domain  Engineering 
Process 


Reuse  Asset 
Management  Process 


Reuse  Program 
Management  Process 


SW  Support 

Software  Documentation 
Management  Process 


Software  Configuration 
Management  Process 


Software  Quality 
Assurance  Process 


Software  Verification 
Process 


Software  Validation 
Process 


Software  Review  Process 


Software  Audit  Process 


Software  Problem 
Resolution  Process 


Adapted  from  WG7  N1111; 
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One  or  more 

12207-based  Outcome/s 
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Activities,  Tasks 

12207  Outcomes 

Activities  and  Tasks 

15288  Outcomes 

Activities,  Tasks 

Building  12207:2008  -  Software  Specific 


Software  Specific  Support  almost  the  same  as  12207  Supporting  Processes 


Technical 


Stakeholder  Reqmts 
Definition  Process 


System  Requirements 
Analysis  Process 


System  Architectural 
Design  Process 


Implementation  Process 


System  Integration 
Process 


System  Qualification 
Testing  Process 


Software  Installation 
Process 


Software  Acceptance 
Support  Process 


Software  Operation 
Process 


Software  Maintenance 
Process 


Software  Disposal 
Process 


Agreement 


Acquisition  Process 


Supply  Process 


Organizational 

Project-Enabling 

Life  Cycle  Model 
Management  Process 


Infrastructure 
Management  Process 


Project  Portfolio 
Management  Process 


Human  Resource 
Management  Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management 
Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


SW  Implementation 


Software  Implementation 
Process 

Software  Requirements 
Analysis  Process 

Software  Architectural 
Design  Process 

Software  Detailed 
Design  Process  1 

Software  Construction 
Process 

Software  Integration 
Process 

Software  Qualification 
Testing  Process 

SW  Reuse 


Domain  Engineering 
Process 


Reuse  Asset 
Management  Process 


Reuse  Program 
Management  Process 


SW  Support 

Software  Documentation 
Management  Process 


Software  Configuration 
Management  Process 


Software  Quality 
Assurance  Process 


Software  Verification 
Process 


Software  Validation 
Process 


Software  Review  Process 


Software  Audit  Process 


Software  Problem 
Resolution  Process 
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Adapted  15288  Outcome/s 

One  or  more 

Blended  12207  &  15288 

One  or  more 

12207-based  Outcome/s 

Activities,  Tasks 

12207  Outcomes 

Activities  and  Tasks 

15288  Outcomes 

Activities,  Tasks 

Building  12207:2008  -  Software  Specific 


12207  Organizational  Processes  for  Reuse  conclude  the  Software  Specific  set 


Technical 


Stakeholder  Reqmts 
Definition  Process 


System  Requirements 
Analysis  Process 


System  Architectural 
Design  Process 


Implementation  Process 


System  Integration 
Process 


System  Qualification 
Testing  Process 


Software  Installation 
Process 


Software  Acceptance 
Support  Process 


Software  Operation 
Process 


Software  Maintenance 
Process 


Software  Disposal 
Process 


Agreement 


Acquisition  Process 


Supply  Process 


Organizational 

Project-Enabling 

Life  Cycle  Model 
Management  Process 


Infrastructure 
Management  Process 


Project  Portfolio 
Management  Process 


Human  Resource 
Management  Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management 
Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


SW  Implementation 


Software  Implementation 
Process 

Software  Requirements 
Analysis  Process 

Software  Architectural 
Design  Process 

Software  Detailed 
Design  Process  1 

Software  Construction 
Process 

Software  Integration 
Process 

Software  Qualification 
Testing  Process 

SW  Reuse 


Domain  Engineering 
Process 


Reuse  Asset 
Management  Process 


Reuse  Program 
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Activities,  Tasks 

12207  Outcomes 

Activities  and  Tasks 

15288  Outcomes 

Activities,  Tasks 

Another  Way  of  Looking  at  It 


Revised  Standards 

■  Front  Matter 

1.  Scope 

2.  Conformance 

3.  Normative  References 

4.  Terms  and  Definitions 

5.  Application  of  this  International  Standard 

6.  System  Life  Cycle  Processes 

7.  Software  Life  Cycle  Processes  {Italicized  indicates  12207  Only} 


The 

A. 

B. 

C. 

D. 

E. 

F. 

G. 

H. 

I. 


12207  Annexes  (12207  and  15288  differ  somewhat  in  format  and  content  here) 

Tailoring  (Normative) 

Process  Reference  Model  (Normative) 

•  1 5504-2  Conformance,  PRM  Lower  Level  Processes  for  Acquisition,  Supply,  Life  Cycle  Model  Management, 
Human  Resource  Management,  and  Software  Operation 

History  and  Rationale  (Informative) 

•  History,  Process  Integration/Constructs  and  Usage,  Relationships,  Process  Definition  Sources 
Process  Alignment  of  12207-15288  {Clause  6}  (Informative) 

Process  Views  (Informative) 

•  Concepts,  and  Process  View  for  Usability  Example 
Some  Example  Process  Descriptions  (Informative) 

Relationship  to  other  IEEE  standards  (Informative) 

Bibliography  (Informative) 

List  of  {IEEE}  participants  (Informative) 

SEIParcncr 
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Aligned  15288  and  12207  Set  Provides 


■  Coordinated  Terms  and  Definitions 

■  Integrated  Process  Structure 

■  Coordinated  Process  Sets 

Backward  compatible 

□  Usable  stand  alone  or  jointly  by  systems  and  software  teams 

System  Context  processes  are  nearly  identical  or  the  12207  processes 
provide  software-appropriate  specializations  of,  or  contribute  to  the 
outcomes  of,  the  corresponding  15288  processes 

Especially  on  Agreement  and  Project  Processes 

■  Common  Conformance/Tailoring 

■  Common  Life  Cycle  Model  and  Stage  Concepts 

■  Free  Guidance  (Annexes  and  Plan  for  TR  24748-1 ) 


Easier  Joint  Use  -  Improved  Efficiency  -  Reduced  Costs 
Common  Acquisition,  Supply  and  Management  Views 


■at  SEifiartncr 


CHSOFT 
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Towards  Full  LCP  Integration 


■  WG7  Study  Group  on  Harmonization  Integration  Strategy  Report 

SC7  Life  Cycle  Process  Harmonization  Advisory  Group  (LCPHAG) 

■  Work  with  SWG5  across  SC7  and  externally  for  analyses  and  recommendations 

■  Model  SC7’s  current  LCPs  and  supporting  standards 

■  Study  Process  Repository  and  Electronic  Publishing  Concepts 
Rigorous  review  of  SC7  Vocabulary  (WG22) 

□  Start  revision  to  15289  (Documentation)  to  reflect  aligned  set. 

■  Some  15288-12207  Integration  Considerations: 

Common  purpose  and  outcomes 
Architecture  of  the  standards 

□  Level  of  prescription  of  activities  and  tasks 

□  Life  cycle  treatments 

□  Application  to  services  and  operations 
Common  verification  and  validation  concepts 
Common  configuration  management  concepts 

□  Alignment  with  other  applicable  standards 

□  Rationalization  of  application  guides 


Source:  WG  7  Nil 03  -  Strategy  for  Integration  Study  Group  Final  Report,  22APR08 


Technical  Software  Services,  Ire. 
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SC7’s  Large  Scale  Harmonization  Efforts 


SWG1 

•  Business 
Planning 

SWG5 

•  Standards 
Management 


Study  Groups,  e.g 

•  Relationships 

*  Integration 


LCPHAG 

•  Modeling 

•  Architectural 
Analysis 

•  Process 
Repository 


Overview  of  the 
SC  7  collection 


Systems  Engineering 


15288 


Tools, 

Methods 


3535,  5806,  5807 
8631,8790,  11411,  14579 


SC7  Legacy  Standards 


12182 


14102,  14471 
15940,  18018 
23026,  29118 
24766 


Tools,  methods 
and  environment 


10746,  13235 
14750,  14752 
14753,  14769 
14771,  15414 
15935,  19500 
19770-2,3 


15437 

15909 

19501 

19739 

8807 

24774 


14568 

15474 

15475 

15476 


Interchange 


Process 
Implementation 
and  Assessment 


Product 

Characteristics 


Draft 


Version  14.2  -  May  2008 


Specifications 


Modeling 
Note:  Italicized  number 


Source:  Adapted  from  ISO/IEC  JTC1/SC7  SWG5  V14.2  briefing  material,  May  2008 


standards  or  TRs  under  development 
Solid  Yellow  =  Aligned  or  in  process; 

Gradient  Yellow  =  Planned  for  alignment _ 
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Harmonization  Across  Collections 

The  State  of  Harmonization  ...  Today 


Topic 

Status 

Remarks 

Terminology  &  Concepts 

Yellow  t 

Shared  BOK,  joint  vocabulary  project  potential  certification  framework 

Quality  management 

Yellow 

IEEE  is  adopting  ISO/IEC  60003  approach 

Testing 

Orange 

Both  IEEE  and  BSI  will  harmonize  with  SC7  processes 

Architecture  description 

Green 

SC7  adopted  IEEE  standard  and  will  harmonize  with  processes. 

Product  quality 

Yellow  t 

ISO/IEC  12116  was  revised  as  25051.  IEEE  will  withdraw  its  standard. 

Life  cycle  processes 

Green 

Systems  engineering 

Green 

Shared  SE  process  standard:  harmonization  with  other  LC  processes  underway 

SW  maintenance 

Green 

Project  to  merge  IEEE  and  ISO  standards  is  completed 

Measurement 

Yellow  t 

IEEE  will  adopt  1 5639  after  its  current  revision.  Some  details  remain. 

Risk  management 

Green 

SC7  adopted  IEEE  standard  and  is  now  extending  it  to  the  systems  level 

Project  management 

Yellow  f 

Project  is  merging  the  incompatible  standards. 

Verification  and  validation 

Fundamentally  different  approaches.  Good  intentions,  but  no  action  yet. 

Configuration  management 

Yellow 

SC7  withdrew  its  standard:  systems  issues  remain.  IEEE  is  about  to  revise 

SW  process  assessment 

Yellow  * 

Harmonization  with  LC  process  standards  is  underway 

Requirements  engineering 

Orange  f 

Joint  project  has  been  approved,  mashup  of  relevant  standards  is  being  prepared 

SW  life  cycle  data 

Yellow  t 

IEEE  is  adopting  15286  to  replace  12207.1 

User  documentation 

Yellow  t 

IEEE  1063  has  been  incorporated  into  26514.  IEEE  will  adopt  it. 

CASE  tools 

Yellow 

Minor  incompatibilities 

Notations 

Harmless 

Distinct  standards  for  distinct  notations 

Internet 

Green 

Shared  standard 

IT  Services.  Management.  Governance 

Yellow 

IEEE  will  adopt  20000  standards 

Specialty  Engineering  (Safety,  Security) 

Orange  <t> 

Unrelated  approaches  will  be  addressed  in  part  by  coordination  revision  of  15026 

Others 

Yellow 

Many  unrelated  standards 

The  State  of  Harmonization  in  1995 


Topic 

Remarks 

Terminology  &  Concepts 

Different  vocabulary  standards 

Quality  management 

ISO:  Driven  down  from  ISO  9001.  IEEE:  traditional  QA  approach. 

Testing 

Orange 

IEEE  standards  unrelated  to  SC7  prooesses. 

Architecture  description 

Harmless 

SC7  didn't  have  architecture  standards 

Product  quality 

Unrelated  standards 

Life  cycle  processes 

Incompatible  standards 

Systems  engineering  process 

Unrelated  standards 

SW  maintenance 

Incompatible  standards 

Measurement 

Yellow 

Unrelated  standards 

Risk  management 

Harmless 

No  standards  at  all 

Project  management 

Incompatible  standards 

Verification  and  validation 

Fundamentally  different  approaches:  minor  incompatibilities  in  details 

Configuration  management 

Incompatible  standards 

SW  prooess  assessment 

Yellow 

Nothing  in  IEEE.  ISO  process  assessment  incompatible  with  ISO  LC. 

Requirements  engineering 

Orange 

IEEE  standards  unrelated  to  SC7  processes 

SW  life  cycle  data 

Incompatible  standards 

User  documentation 

Incompatible  standards 

CASE  tools 

Yellow 

Minor  incompatibilities 

Notations 

Harmless 

Distinct  standards  for  distinct  notations 

Internet 

Harmless 

No  standards 

IT  Services.  Management.  Governance 

Harmless 

No  standards 

Specialty  Engineering  (Safety.  Security) 

Orange 

Unrelated  approaches 

Others 

Yellow 

Many  unrelated  standards 

Source:  07N3997  2008-05  IEEE-CS  Liaison  Report  to  SC7-  J.  Moore,  MITRE 
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IEEE  CS  May  2008 
Status  Report  to  SC7 

Stoplight  charts 
show  marked 
improvement 
between  the 
IEEE  and  SC7 
Standards 
Collections 
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Harmonization  Benefits  Summary 


Alignment 

■  Achieves  short  term  objectives 

■  Maintains  backward  compatibility 

■  Starts  disparate  users  towards  goal 

Integration 

■  Tackles  the  ‘religious’  issues 

□  Technical  and  Political 

■  Achieves  long  term  goals  in  a  set 

Large  Scale  Harmonization 

■  Solves  big  picture  issues  within  and 
across  SDOs 


Each  Level  Brings  You 

■  Easier  process  definition  and 
implementation 

■  Better  team  communication 
and  integration 

■  Improved  performance  at 
lower  cost 

■  Increased  benefit  and 
usefulness  of  implementing 
these  standards  in  your 
organization 


Eases  Your  Integration,  Management,  and  Acquisition  Burden  | 


snpi  *~2~  £EJftar'f>cr 
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Questions? 


41 


For  More  Information  Contact 


Teresa  ‘Terry’  Doran 
TECHSOFT 

31  West  Garden  Street,  Suite  100 
Pensacola,  FL  32502-5685 
Internet:  www.techsoft.com 


NY  Office  Tel:  1  631-266-2191 
Email:  ts 


ISO/IEC/IEEE  12207  Project  Editor 

15288-12207-24748  Editorial  Team  Member 

IEEE  Std  1220™-2005  Project  Editor  (aka  ISO/IEC  26702:2007) 

ISO/IEC  JTC1/SC7  Life  Cycle  Process  Advisory  Group  Chair 
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Abbreviations  - 1 


ANSI 

CMMI 

CMU 

IEC 

IEEE 

IEEE  CS 

INCOSE 

ISO 

IT 

JTC1 

LCP 

NWIP 

OPA 

OPD 

SC 

SG 


-  American  National  Standards  Institute 

-  Capability  Maturity  Model  Integration 

-  Carnegie  Mellon  University 

-  International  Electrotechnical  Commission 

-  Institute  of  Electrical  and  Electronics  Engineers 

-  IEEE  Computer  Society 

-  International  Council  on  Systems  Engineering 

-  International  Organization  for  Standardization 

-  Information  Technology 

-  ISO/IEC  Joint  Technical  Committee  1:  Information  Technology 

-  life  cycle  process 

-  new  work  item  proposal 

-  organizational  process  assets 

-  organizational  process  definition 

-  subcommittee 

-  study  group 


ANSI 

CMMI 

CMU 

IEC 

IEEE 

IEEE  CS 

INCOSE 

ISO 

IT 

JTC1 

LCP 

NWIP 

OPA 

OPD 

SC 

SG 


=-«  =  sei  -<*■ 
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Technical  Software  Services,  Ire. 


Abbreviations  -  2 

SC7  -  ISO/IEC  JTC1  SC  7:  Software  and  Systems  Engineering 

SE  -  systems  engineering 

SEI  -  Software  Engineering  Institute  (at  CMU) 

S2ESC  -  Software  and  Systems  Engineering  Standards  Committee  (IEEE  CS) 

SEP  -  SE  process 

SWE  -  software  engineering 

SWG  -  special  WG 

WG  -  working  group 

WG7  -  ISO/IEC  JTC1  SC7  WG  7:  Life  Cycle  Management 
VSE  -  very  small  enterprise 


SEIParcnor 
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References  - 1 


For  ISO  and  ISO/IEC  Standards  (Current  and  Withdrawn): 

h tto ://www. iso. org/iso/iso  cataloguejitm 

1)  ISO  9001 :2005,  Quality  management  systems  —  Requirements 

2)  ISO/IEC  12207:2008,  Systems  and  software  engineering  — 
Software  life  cycle  processes 

3)  ISO/IEC  15288:2008,  Systems  and  software  engineering  — 
System  life  cycle  processes 

For  ISO/IEC  documents  and  in-process  standards  and 
technical  reports  (TRs):  http ://www. itc1-sc7. org/ 

4)  SC7  N4143:  ISO/IEC  DTR  24748.2:2009,  Systems  and  software 
engineering  —  Guide  for  life  cycle  management 
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References  -  2 


For  IEEE  Standards: 

http://www.ieee.org/web/standards/home/index.html 

IEEE  Std  1 220™-2005,  IEEE  Standard  for  Application  and 
Management  of  the  Systems  Engineering  Process 

Or  related  information: 

http://standards.computer.org/s2esc/ 

IEEE  CS  Software  and  Systems  Engineering  Standards  Committee 
-  for  on-going  SE/SW  standards  activities 

h ttp://pascal. computer. org/sev  displav/index. action 

SEVOCAB:  An  IEEE  CS  and  ISO/IEC  JTC  1/SC7  project,  SEVOCAB 
includes  definitions  from  international  standards;  This  database  is 
issued  periodically  as  a  formal,  published  International  Standard 
(ISO/IEC  24765)  reflecting  a  "snapshot"  of  the  database. 
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Systems  and  Software  Life  Cycle 
Process  Standards:  Foundation 
for  Integrated  Systems  and 
Software  Engineering 


For:  NDIA  Systems  Engineering  Conference 

Date:  23  October  2008 
Presented  By:  Teresa  Doran 

TECHSOFT  NDIA-Brief-0001 

TSDoran-NDIA-SE  23OCT08  vl.O 
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Overview 


■  Purpose 

■  TECHSOFT 

■  Standards-based  Process  Improvement  Success 

■  Why  Harmonize? 

□  Issues 

□  Impacts  to  you 

■  SE/SW  LCP  Alignment  and  Integration 

□  Path 

□  Concepts 

□  Where  we  are  today 

How  we  got  here  -  Key  changes  in  15288  &  12207 

■  Large  Scale  Harmonization 

■  Benefits  Summary 

ap  $6if\wmcr 

IIIechsoft 
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Show  how  the  key  changes  in  the  alignment  of  a 
foundational  systems/software  standards  set 
(ISO/IEC/IEEE  15288  and  ISO/IEC/IEEE  12207)  facilitates 
integrated  systems  and  software  engineering,  project 
management,  and  acquisition 
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Who  We  Are 

■  Founded  in  1990 

■  Based  in  Pensacola,  Florida 

□  Presence  in  Charleston,  SC 

■  Primarily,  a  DoD  Contractor 

■  Experienced  Staff 

High  %  Masters  level  personnel 

□  Majority  with  Security 
Clearances 

SEI-Authorized  CMMI®  Lead 
Appraisers 

SEI-Authorized  CMMI® 
Instructors 

□  International  SE/SW  Standards 
Expertise 
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What  We  Do 

Systems  &  Software 
Development 

Database  Applications 

Security  /  IA 

Web  Development 

Network  Engineering/Hosting 

Training 

Process  Engineering/Process 
Improvement 

□  CMMI® 

□  SEI  Partner 


ip  SElftanncr 
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Standards-based  Process  Improvement 

Example  of  a  Successful  Approach 


srm/AR 


Systems  Center 

Charleston 


Process  Improvement  and 


•^svstea^ 


Vision 

-  Develop  and  maintain  a  World  Class  Systems  Engineering 
Organization 

Approach 

-  Achieve  Command-wide  operational  consistency 

-  Based  on  ISO  15288-  systems  engineering 

-  Based  on  ISO  12207  -  software  engineering 

-  Measure  using  best  practices  of  CMMI® 

Goals 

-  CMMI®  Maturity  Level  2  by  April,  2005 

-  CMMI®  Maturity  Level  3  by  April,  2007 

Both  Goals  attained  on  schedule 
1st  SPAWAR  Systems  Center  to  Achieve  ML2  and  ML3 
New  Goal:  Maturity  Level  4  &  5 

Approved  for  public  release;  distribution  is  unlirrited  (15  JUL2008) 


MATURITY 


Source:  N65236-ENGOPS-BRIEF-0068-1.1,  Standardization  of  Systems  Engineering  &  Project  Management  Using  CMMI,  M.T.  Kutch,  Jr.,  17JUL08 
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Full  OPD,  But  Today’s  Focus:  15288/1 2207 


This  SSC  has 
15288  and 
12207-based 
SE/SWE 
Technical 
Processes 


spmaR 

Y 


Systems  Center 

Charleston 


SSC-0  SE  Revitalization  Plan 
Aligned  with  DoD  SE  Revitalization 


Elements  of  SSC-C  SE  Revitalization 


Policy  /  Guidance 

■ 

Training  /  Education 

■ 

Assessment  &  Support 

Intro  to  PI  WBT 


CM  Ml®  Level  2 


SE  101  WBT 


CM  Ml®  Level  3 


SSC-C  JUU-Mcilnt 
Process  Manual 


SE  Fundamentals 


CM  Ml®  Level  4/5 


SE  for  Managers 


Project  Reviews 


EPO  Website 


Project  &  Process 
Workshop 


Balanced  Scorecard 


ePIan  Builder 


Intro  to  Software  Engr. 


Lean  Six  Sigma 


CJ  Underway 

I  Completed/Ongoing 


Architecture  Dev.  WBT 


Integrated  Product 
Teams 


Process  Asset  Library 


in  i;  mi  i|i  ill  i:  111  i!:  mi  M.  mi '  |  || — | - rrn - 

Certification/Degrees 


IT  Tools 


IIechsoft 


With  Extensive 
OPA  Support 


Source:  N65236-ENGOPS-BRIEF-0048-1.2,  Tools  and  Resources  to  Enable 
Systems  Engineering  Improvement,  M.T.  Kutch,  Jr.  &  M.  Knox,  NOV07 
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Standard 

Processes 


*  Process 
Manuals 

*  SOPb 

*  Sample 
Documents 

*  Templates 


Project  Lite 
Cycles 


•  Waterfall 
■  Spiral 

*  Incremental 


Library  of 
Process-Related 
Documentation 


•  Sample 
Documents 

•Searchable 

Database 


Tailoring 

Criteria 


•  Policy 

•  Sample 
Documents 


•  Protect  Data 

•  Corporate  Data 

•  Cost 


IIechsoft 


So  what’s  the  problem  with  15288  and  12207 


ISO/IEC  15288:2002 


Enterprise  Processes 

Enterprise  Environment 
Management  Process 


Investment  Management 
Process 


System  Life  Cycle  Processes 
Management  Process 


Resource  Management 
Process 


Quality  Management 
Process 

Agreement  Processes 


Project 

Processes 

Project  Planning 
Process 


Project  Assessment 
Process 


Project  Control 
Process 


Decision-making 

Process 


Risk  Management 
Process 


Configuration 
Management  Process 


Information 
Management  Process 


Technical  Processes 

Stakeholder  Requirements 
Definition  Process 

Requirements  Analysis 
Process 

Architectural  Design 
Process 

Implementation  Process 

Integration  Process 

Verification  Process 

Transition  Process 

Validation  Process 

Operation  Process 

Maintenance  Process 

Disposal  Process 

Using  Them  Together! 

•  Conflicting  terms  and  definitions 

•  Overlapping,  yet  distinct  processes 

•  Different  process  architectures 

•  Different  levels  of  prescription 


ISO/IEC  12207:1995 


5.  PRMtARY 

UFE  CYCLE  PROCESSES 


5.1  Acquisition 


5.2  Si^piy 


5.3 

Development 


5.4 


5.5 


5.  SUPPORTING 
LIFE  CYCLE  PROCESSES 


6.1  Documentation 


6_2  Configuration 


=npi  SEIftOTocr 
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Unintegrated  12207  amendments 
from  2002  and  2004  are  difficult  to 
use  and  also  not  adopted  by  IEEE 
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Why  You  Should  Care 


■  Leverage  the  Commonalties 

Identify  and  explain  the  differences 
Use  the  interfaces 

■  Promote  Communication  and  Team  Integration 

Identify  strengths,  views,  and  appropriate  focused  implementations 

□  Reduce  us/them,  finger-pointing,  stove-piping 

■  Improve  Resource  Performance 

□  Personnel,  Processes,  Tools,  Services 

■  Lower  Costs 

Reduce  redundancy  and  inefficiency 


Benefits  of  Standards  Harmonization 

Supports  Integration,  Facilitates  Management,  Simplifies  Acquisition 


Te 
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15288-12207  Harmonization  Path 


Harmonization  Study  Group 

Ikjinri  co  the  Advisory  (i  rou  p  Marti  real  2t>03 


Doug  Tfile-le- 
WG  7  Converter 


Studies 


Implementation  hits  a  snag 


ISO/EC  JTC  1/SC  7/WG7  N0&42 
2006  04  16 


‘Harmonization  Lite” 
— A  Proposal 


_  _  James  W.  Moore,  CSDP 

soaEiY  The  MITRE  Corporation 

IEEE  Computer  Society  Liaison  to  ISO/IEC  JTC  1/SC  7 
J3  mes.W.MoorG@ioeo.org 

ft  6.1, 17  April  2005 

lauthor  and  do  not 
Is  sponsors. 


The  opinions  cc 
neeessariT^ 


’05-’07 


Eat  that 
elephant 
one  bite 
at  a  time! 


ISO/IEC  JTC  1/SC  7/WG  7  N0868 
2005-05-27 


Harmonization  revised  concept 


11/06/2006 


Align  -  Publicize  -  Integrate 


TSDoran-NDIA-SE  23OCT08  vl.O 


Concept  for  the  Harmonized  Set 


Source:  ISO/IEC  JTC1/SC7  WG7  N01025  Briefing  Material,  24MAY07 
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Where  We  Are  Today 


Nearly  identical  process  models 


System 

Level 

Processes 


INTERNATIONAL  ISO/IEC 

STANDARD  15288 


INTERNATIONAL  ISO/IEC 

STANDARD  12207 


IEEE 

Std  15288-2003 


IEEE 

Std  12207-2003 


Second  edition 
2i]i]a-D2-[H 


£econd  ecition 
2GQa-D2-D1 


Systems  and  software  engineering  — 
System  life  cycle  processes 

Ingenrene  ties  syctemes  et  du  iogicief —  Processus  du  eye le  de  we 
du  eysteme 


Systems  and  software  engineering  — 
Software  life  cycle  processes 

ingenrene  des  syaf ernes  et  du  fogiciei  —  Processus  du  cycle  de  w'e 
du  fogidel 


System 
Processes 
Specialized 
To  Software 
and 

Software- 

Specific 

Processes 


Life  Cycle  Concepts 
Process  Concepts 
LC  Models,  Stages 

ISO/IEC  JTC  1/SC  7 

Date  2008-08-11 

ISO/IEC  DTR  24748-1 

[)  ^  ISO  IEC  JTC  1/SC  7/WG  7  N1 1 40 

Secretarial;  SCC 

LC  Adaptation 
Domains,  Disciplines, 

&  Specialties 

Prior  Version  Transition 

Systems  and  software  engineering - Guide  for  life  cycle 

management 

Ingeniene  systemes  et  logtctel - Guide  pour  geatidn  du  cycle  de  we 

SH  SElftartncr 

IIIechsoft 

It  ii  the  intention  of  this  pioject  to  cieate  a  Technical  Report  of  Type  3  that  may  be  made  freely  available  in  accordance 
with  the  provisions  of  JTC  1  N  7269  and  Sendai  Revolution  32.  In  paiticulat  the  document  has  the  following 
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Relations  of  Process  Constructs  among  ISO/IEC  12207:1995 
and  its  Amendments,  15288:2002, 15288:2008  &  12207:2008 


12207/15288:2008  Process  Constructs 


Processes  require  a  purpose  and  outcome.  All 
processes  have  at  least  one  activity.  The  processes, 
with  their  statements  of  purpose  and  outcomes, 
constitute  a  Process  Reference  Model  (PRM). 


Activities  are  constructs  for  grouping  together 
related  tasks.  The  activities  provide  a  means  to  look 
at  related  tasks  within  the  process  to  improve 
understanding  and  communication  of  the  process.  If 
an  activity  is  cohesive  enough,  it  can  be  converted 
to  a  (lower  level)  process  by  defining  a  purpose  and 
a  set  of  outcomes. 


A  task  is  a  detailed  provision  for  implementation  of  a 
process.  It  may  be  a  requirement  (“shall”),  a 
recommendation  (“should”),  or  a  permission  (“may”). 


Notes  are  used  when  there  is  a  need  for  explanatory 
information  to  better  describe  the  intent  or 
mechanics  of  a  process.  Notes  provide  insight 
regarding  potential  implementation  or  areas  of 
applicability  such  as  lists,  examples  and  other 
considerations. 


Adapted  from  ISO/IEC  JTC1/SC7  WG7  N1025  briefing  material 


Normative 


►  Informative 


/ 
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The  Life  Cycle  Processes  of  15288:2002 


Agreement 


Acquisition  Process 

Supply  Process 

Enterprise 

Enterprise  Environment 
Management  Process 


Investment  Management 
Process 


System  LC  Processes 
Management  Process 


Resource  Management 
Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment 
Process 


Project  Control  Process 


Decision-Making  Process 


Risk  Management  Process 


Configuration 
Management  Process 


Information  Management 
Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 


Requirements  Analysis 
Process 


Architectural  Design 
Process 


Implementation  Process 


Integration  Process 


Verification  Process 


Transition  Process 


Validation  Process 


Operation  Process 


Maintenance  Process 


Disposal  Process 


Source:  WG7  Nil  11;  Adapted  by  Jim  Moore,  MITRE  Corporation  from  chart  by  Anatol  Kark,  National  Research  Council,  Canada 

14 
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Building  15288:2008  -  Activities  and  Tasks 

■\ 


Activity-Task  allocation  is  new  to  15288:2008 
Provides  structural  alignment  with  12207 


Agreement 


Acquisition  Process 


Supply  Process 


Enterprise 


Enterprise  Environment 
Management  Process 


Investment  Management 
Process 


System  LC  Processes 
Management  Process 


Resource  Management 
Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment 
Process 


Project  Control  Process 


Decision-Making  Process 


Risk  Management  Process 


Configuration 
Management  Process 


Information  Management 
Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 


Requirements  Analysis 
Process 


Architectural  Design 
Process 


Implementation  Process 


Integration  Process 


Verification  Process 


Transition  Process 


Validation  Process 


Operation  Process 


Maintenance  Process 


Disposal  Process 


0..* 

Process 

Name,  Purpose, 
Outcome(s) 

T) 

A  * 

C. 

Activity 


Name 


1..* 


Task 


0..* 


Note 


> 


Normative 


J 


Informative 


Adapted  from  WG7  N1111;  Source:  Jim  Moore,  MITRE  Corporation  and  Anatol  Kark, 
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Tb 


■  SEIf^encr 


Building  15288:2008  -  Technical  Processes 


15288:2008  has  the  same  set  of  technical  processes  as  15288:2002 


Agreement 


Acquisition  Process 

Supply  Process 

Enterprise 

Enterprise  Environment 
Management  Process 


Investment  Management 
Process 


System  LC  Processes 
Management  Process 


Resource  Management 
Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment 
Process 


Project  Control  Process 


Decision-Making  Process 


Risk  Management  Process 


Configuration 
Management  Process 


Information  Management 
Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 


Requirements  Analysis 
Process 


Architectural  Design 
Process 


Implementation  Process 


Integration  Process 


Verification  Process 


Transition  Process 


Validation  Process 


Operation  Process 


Maintenance  Process 


Disposal  Process 


Technical 

Stakeholder  Reqmts 
Definition  Process 

Requirements  Analysis 
Process 

Architectural  Design 
Process 

Implementation  Process 

Integration  Process 

Verification  Process 

Transition  Process 

Validation  Process 

Operation  Process 

Maintenance  Process 

Disposal  Process 

Source:  WG7  Nil  11;  Adapted  by  Jim  Moore,  MITRE  Corporation  from  chart  by  Anatol  Kark,  National  Research  Council,  Canada 
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Building  15288:2008  -  Project  Processes 


15288:2008  has  a  similar  set  of  project  processes  as  15288:2002 


Agreement 


Acquisition  Process 


Supply  Process 


Enterprise 

Enterprise  Environment 
Management  Process 


Investment  Management 
Process 


System  LC  Processes 
Management  Process 


Resource  Management 
Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management  Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 


Requirements  Analysis 
Process 


Architectural  Design 
Process 


Implementation  Process 


Integration  Process 


Verification  Process 


Transition  Process 


Validation  Process 


Operation  Process 


Maintenance  Process 


Disposal  Process 


Source:  WG7  Nil  11;  Adapted  by  Jim  Moore,  MITRE  Corporation  from  chart  by  Anatol  Kark,  National  Research  Council,  Canada 
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Building  15288:2008  -  Project-Enabling  Processes 

15288:2008  has  a  similar  set  of  project-enabling  processes  as  15288:2002 


Agreement 


Acquisition  Process 


Supply  Process 


Organizational 

Project-Enabling 


Life  Cycle  Model 
Management  Process 


Infrastructure 
Management  Process 


Project  Portfolio 
Management  Process 


Human  Resource 
Management  Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management  Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 


Requirements  Analysis 
Process 


Architectural  Design 
Process 


Implementation  Process 


Integration  Process 


Verification  Process 


Transition  Process 


Validation  Process 


Operation  Process 


Maintenance  Process 


Disposal  Process 


Source:  WG7  Nil  11;  Adapted  by  Jim  Moore,  MITRE  Corporation  from  chart  by  Anatol  Kark,  National  Research  Council,  Canada 
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Building  15288:2008  -  Agreement  Processes 

15288:2008  has  the  same  set  of  agreement  processes  as  15288:2002 


Agreement 

Agreement 

Acquisition  Process 

Acquisition  Process 

Supply  Process 

Supply  Process 

Organizational 

Project-Enabling 


Life  Cycle  Model 
Management  Process 


Infrastructure 
Management  Process 


Project  Portfolio 
Management  Process 


Human  Resource 
Management  Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management  Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 


Requirements  Analysis 
Process 


Architectural  Design 
Process 


Implementation  Process 


Integration  Process 


Verification  Process 


Transition  Process 


Validation  Process 


Operation  Process 


Maintenance  Process 


Disposal  Process 


Source:  WG7  Nil  11;  Adapted  by  Jim  Moore,  MITRE  Corporation  from  chart  by  Anatol  Kark,  National  Research  Council,  Canada 
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The  Life  Cycle  Processes  of  15288:2008 


Agreement 


Acquisition  Process 


Supply  Process 


Organizational 

Project-Enabling 


Life  Cycle  Model 
Management  Process 


Infrastructure 
Management  Process 


Project  Portfolio 
Management  Process 


Human  Resource 
Management  Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management  Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


Technical 

Stakeholder  Reqmts 
Definition  Process 

Requirements  Analysis 
Process 

Architectural  Design 
Process 

Implementation  Process 

Integration  Process 

Verification  Process 

Transition  Process 

Validation  Process 

Operation  Process 

Maintenance  Process 

Disposal  Process 

Source:  WG7  Nil  11;  Adapted  by  Jim  Moore,  MITRE  Corporation  from  chart  by  Anatol  Kark,  National  Research  Council,  Canada 
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The  Life  Cycle  Processes  of  12207:1995 


5L  PRIMARY 

UFE  CYCLE  PROCESSES 


5.1  Acqusition 


5 1  Supply 


6L  SUPPORTING 
UFE  CYCLE  PROCESSES 


6.1  Docianentation 


6.2  Conflgurafion 
Management 


6.8  Problem  Resolution 


The  Familiar  1995 
LCP  Categories 
Process  Structure 
and  Titles 


7.  ORGANIZATIONAL  UFE  CYCLE  PROCESSES 


7.1  Management 


7_2  Infrastructure 


7.3  Improvement 


7.4  Training 


Adapted  from  WG7  N1111  briefing  material 
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S^M  $EI 'VkCiK:! 

Iechsoft 


Technical  Software  Services,  Ire. 


The  Life  Cycle  Processes  of  12207:1995 


Primary 


Acquisition  Process 


Supply  Process 


Organizational 


Improvement  Process 


Infrastructure  Process 


Management  Process 


Training  Process 


Primarily  Primarily 

organization-  project- 

oriented  oriented 


Development  Process 


System  Requirements 
Analysis 


System  Architectural 
Design 


System  Integration 


System  Qualification 
Testing 


Software  Installation 


Software  Acceptance 
Support 


Process  Implementation 


Software  Requirements 
Analysis 


Software  Architectural 
Design 


Software  Detailed  Design 


Software  Coding  & 
Testing 


Software  Integration 


Software  Qualification 
Testing 


Operation  Process 


Maintenance  Process 


Adapted  from  WG7  N1111;  Source:  Jim  Moore,  MITRE  Corporation 
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]  Box  with  dashed  border  \ 
i  was  an  Activity  in  1995  j 


Supporting 

Documentation 
Management  Process 


Configuration 
Management  Process 


Quality  Assurance 
Process 


Verification  Process 


Validation  Process 


Joint  Review  Process 


Audit  Process 


Problem  Resolution 
Process 


SEIFfrmcf 

CHSDFT 

Technical  Software  Services,  Inc. 


12207  Amd. 1:2002  and  Amd. 2:2004 


■  Defined  a  Process  Reference  Model  (PRM)  for  12207 

□  Process  Name,  Purpose,  and  Outcomes 

■  Restructured  processes  to  provide  higher  granularity 

Introduced  sub-processes  (e.g  based  on  Development  activities) 

Improvement,  Human  Resource,  Acquisition,  Supply,  Development, 
Operation,  Management 

■  Introduced  extensions,  elaborations  and  new  processes 

e.g.  to  better  support  process  assessment  (15504-2), 
usability(13407),  measurement  (15939),  product  evaluation(14598), 
and  reuse/asset  management  (IEEE  1517) 

■  Added  activities  and  tasks  for  8  new  processes 

■  Made  some  corrections 


Generally  aligned  and  incorporated  in  body  of  revised  12207 
Several  sub-processes  allocated  as  lower-level  PRM  only  processes 


■jp  SEIfianncr 

CHSQFT 


Technical  Software  Services,  Ire. 
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The  Life  Cycle  Processes  of  12207:2008 


Agreement 


Acquisition  Process 


Supply  Process 


Organizational 

Project-Enabling 

Life  Cycle  Model 
Management  Process 


Infrastructure 
Management  Process 


Project  Portfolio 
Management  Process 


Human  Resource 
Management  Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management  Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 


System  Requirements 
Analysis  Process 


System  Architectural 
Design  Process 


Implementation  Process 


System  Integration 
Process 


System  Qualification 
Testing  Process 


Software  Installation 
Process 


Software  Acceptance 
Support  Process 


Software  Operation 
Process 


Software  Maintenance 
Process 


Software  Disposal 
Process 


SW  Implementation 


Software  Implementation 
Process 

Software  Requirements 
Analysis  Process 

Software  Architectural 
Design  Process 

Software  Detailed 
Design  Process 

Software  Construction 
Process 

Software  Integration 
Process 

Software  Qualification 
Testing  Process 

SW  Reuse 


Domain  Engineering 
Process 


Reuse  Asset 
Management  Process 


Reuse  Program 
Management  Process 


Adapted  from  WG7  N1111;  Source:  Jim  Moore,  MITRE  Corporation 
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SW  Support 

Software  Documentation 
Management  Process 


Software  Configuration 
Management  Process 


Software  Quality 
Assurance  Process 


Software  Verification 
Process 


Software  Validation 
Process 


Software  Review  Process 


Software  Audit  Process 


Software  Problem 
Resolution  Process 


SSFtetr m 

CHSDFT 

TecfanEcai  Software  Services.  Inc.. 


Building  12207:2008  -  System  Context 

Structural  alignment  with  15288  system  level  categories 

f 


Agreement 


Acquisition  Process 


Supply  Process 


Organizational 

Project-Enabling 


Life  Cycle  Model 
Management  Process 


Infrastructure 
Management  Process 


Project  Portfolio 
Management  Process 


Human  Resource 
Management  Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management  Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


Adapted  from  WG7  N1111; 
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Technical 

Stakeholder  Reqmts 
Definition  Process 

System  Requirements 
Analysis  Process 

System  Architectural 
Design  Process 

Implementation  Process 

System  Integration 
Process 

System  Qualification 
Testing  Process 

Software  Installation 
Process 

Software  Acceptance 
Support  Process 

Software  Operation 
Process 

Software  Maintenance 
Process 

Software  Disposal 
Process 

SW  Implementation 


Software  Implementation 
Process 


Software  Requirements 
Analysis  Process 


Software  Architectural 
Design  Process 


Software  Detailed 
Design  Process 


Software  Construction 
Process 


Software  Integration 
Process 


Software  Qualification 
Testing  Process 


SW  Reuse 


Domain  Engineering 
Process 


Reuse  Asset 
Management  Process 


Reuse  Program 
Management  Process 


SW  Support 


Software  Documentation 
Management  Process 


Software  Configuration 
Management  Process 


Software  Quality 
Assurance  Process 


Software  Verification 
Process 


Software  Validation 
Process 


Software  Review  Process 


Software  Audit  Process 


Software  Problem 
Resolution  Process 


SEIFfrmcr 

IIechsoft 

Technical  Software  Services,  Inc.. 


Building  12207:2008  -  System  Context 


System  Context  Processes  based  on  15288  Processes 


Agreement 


Acquisition  Process 

Supply  Process 

Organizational 

Project-Enabling 


Life  Cycle  Model 
Management  Process 

Infrastructure 
Management  Process 

Project  Portfolio 
Management  Process 

Human  Resource 
Management  Process 

Quality  Management 
Process 

Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management  Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 

System  Requirements 
Analysis  Process 

System  Architectural 
Design  Process 

Implementation  Process 

System  Integration 
Process 

System  Qualification 
Testing  Process 

Software  Installation 
Process 

Software  Acceptance 
Support  Process 

Software  Operation 
Process 

Software  Maintenance 
Process 

Software  Disposal 
Process 


SW  Implementation 


Software  Implementation 
Process 

Software  Requirements 
Analysis  Process 

Software  Architectural 
Design  Process 

Software  Detailed 
Design  Process 

Software  Construction 
Process 

Software  Integration 
Process 

Software  Qualification 
Testing  Process 

SW  Reuse 


Domain  Engineering 
Process 


Reuse  Asset 
Management  Process 


Reuse  Program 
Management  Process 


Adapted  from  WG7  N1111; 
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SW  Support 

Software  Documentation 
Management  Process 


Software  Configuration 
Management  Process 


Software  Quality 
Assurance  Process 


Software  Verification 
Process 


Software  Validation 
Process 


Software  Review  Process 


Software  Audit  Process 


Software  Problem 
Resolution  Process 


SSFtetr m 

CHSDFT 

Tecfanka!  Software  Services.  Inc.. 


Building  12207:2008  -  System  Context 


Include  12207  Organizational  Processes:  Improvement,  Infrastructure, 


Human  Resource/Training,  Management 


Agreement 


Acquisition  Process 

Supply  Process 

Organizational 

Project-Enabling 


Life  Cycle  Model 
;  Management  Process 

Infrastructure 
Management  Process 

Project  Portfolio 
Management  Process 

Human  Resource 
Management  Process 

Quality  Management 
Process 

Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management  Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 

System  Requirements 
Analysis  Process 

System  Architectural 
Design  Process 

Implementation  Process 

System  Integration 
Process 

System  Qualification 
Testing  Process 

Software  Installation 
Process 

Software  Acceptance 
Support  Process 

Software  Operation 
Process 

Software  Maintenance 
Process 

Software  Disposal 
Process 


< 


SW  Implementation 


Software  Implementation 
Process 

Software  Requirements 
Analysis  Process 

Software  Architectural 
Design  Process 

Software  Detailed 
Design  Process 

Software  Construction 
Process 

Software  Integration 
Process 

Software  Qualification 
Testing  Process 

SW  Reuse 


Domain  Engineering 
Process 


Reuse  Asset 
Management  Process 


Reuse  Program 
Management  Process 


SW  Support 

Software  Documentation 
Management  Process 


Software  Configuration 
Management  Process 


Software  Quality 
Assurance  Process 


Software  Verification 
Process 


Software  Validation 
Process 


Software  Review  Process 


Software  Audit  Process 


Software  Problem 
Resolution  Process 


Adapted  from  WG7  N1111; 

Adapted  15288  Outcome/s 

One  or  more 

Blended  12207  &  15288 

One  or  more 

12207-based  Outcome/s 
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Activities,  Tasks 

12207  Outcomes 

Activities  and  Tasks 

15288  Outcomes 

Activities,  Tasks 

Building  12207:2008  -  System  Context 


Risk  Management  from  16085  and  Measurement  from  15939  are  added 


Agreement 


Acquisition  Process 

Supply  Process 

Organizational 

Project-Enabling 


Life  Cycle  Model 
;  Management  Process 

Infrastructure 
Management  Process 

Project  Portfolio 
Management  Process 

Human  Resource 
Management  Process 

Quality  Management 
Process 

Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management  Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 

System  Requirements 
Analysis  Process 

System  Architectural 
Design  Process 

Implementation  Process 

System  Integration 
Process 

System  Qualification 
Testing  Process 

Software  Installation 
Process 

Software  Acceptance 
Support  Process 

Software  Operation 
Process 

Software  Maintenance 
Process 

Software  Disposal 
Process 


SW  Implementation 


Software  Implementation 
Process 

Software  Requirements 
Analysis  Process 

Software  Architectural 
Design  Process 

Software  Detailed 
Design  Process 

Software  Construction 
Process 

Software  Integration 
Process 

Software  Qualification 
Testing  Process 

SW  Reuse 


Domain  Engineering 
Process 


Reuse  Asset 
Management  Process 


Reuse  Program 
Management  Process 


SW  Support 

Software  Documentation 
Management  Process 


Software  Configuration 
Management  Process 


Software  Quality 
Assurance  Process 


Software  Verification 
Process 


Software  Validation 
Process 


Software  Review  Process 


Software  Audit  Process 


Software  Problem 
Resolution  Process 


Adapted  from  WG7  N1111; 

Adapted  15288  Outcome/s 

One  or  more 

Blended  12207  &  15288 

One  or  more 

12207-based  Outcome/s 
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Activities,  Tasks 

12207  Outcomes 

Activities  and  Tasks 

15288  Outcomes 

Activities,  Tasks 

Building  12207:2008  -  System  Context 


Risk  Management  and  Measurement  are  now  almost  identical  to  15288 


12207  Acquisition  and  Supply  are  blended  with  15288  Agreement  Processes 


Agreement 


Acquisition  Process 

Supply  Process 

Organizational 

Project-Enabling 

Life  Cycle  Model 
Management  Process 


Infrastructure 
Management  Process 


Project  Portfolio 
Management  Process 


Human  Resource 
Management  Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management  Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 

System  Requirements 
Analysis  Process 

System  Architectural 
Design  Process 

Implementation  Process 

System  Integration 
Process 

System  Qualification 
Testing  Process 

Software  Installation 
Process 

Software  Acceptance 
Support  Process 

Software  Operation 
Process 

Software  Maintenance 
Process 

Software  Disposal 
Process 


SW  Implementation 


Software  Implementation 
Process 

Software  Requirements 
Analysis  Process 

Software  Architectural 
Design  Process 

Software  Detailed 
Design  Process 

Software  Construction 
Process 

Software  Integration 
Process 

Software  Qualification 
Testing  Process 

SW  Reuse 


Domain  Engineering 
Process 


Reuse  Asset 
Management  Process 


Reuse  Program 
Management  Process 


SW  Support 

Software  Documentation 
Management  Process 


Software  Configuration 
Management  Process 


Software  Quality 
Assurance  Process 


Software  Verification 
Process 


Software  Validation 
Process 


Software  Review  Process 


Software  Audit  Process 


Software  Problem 
Resolution  Process 


Adapted  from  WG7  N1111; 

Adapted  15288  Outcome/s 

One  or  more 

Blended  12207  &  15288 

One  or  more 

12207-based  Outcome/s 
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Activities,  Tasks 

12207  Outcomes 

Activities  and  Tasks 

15288  Outcomes 

Activities,  Tasks 

Building  12207:2008  -  System  and  Software 


Development  Activities  form  System  Context  and  Software  Specific  Processes 


Agreement 


Acquisition  Process 

Supply  Process 

Organizational 

Project-Enabling 

Life  Cycle  Model 
Management  Process 


Infrastructure 
Management  Process 


Project  Portfolio 
Management  Process 


Human  Resource 
Management  Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management  Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 

System  Requirements 
Analysis  Process 

System  Architectural 
Design  Process 

Implementation  Process 

System  Integration 
Process 

System  Qualification 
Testing  Process 

Software  Installation 
Process 

Software  Acceptance 
Support  Process 

Software  Operation 
Process 

Software  Maintenance 
Process 

Software  Disposal 
Process 


SW  Implementation 


Software  Implementation 
Process 

Software  Requirements 
Analysis  Process 

i 

Software  Architectural 
Design  Process 

Software  Detailed 
Design  Process 

Software  Construction 
Process 

Software  Integration 
Process 

i 

Software  Qualification 
Testing  Process 

SW  Reuse 


Domain  Engineering 
Process 


Reuse  Asset 
Management  Process 


Reuse  Program 
Management  Process 


SW  Support 

Software  Documentation 
Management  Process 


Software  Configuration 
Management  Process 


Software  Quality 
Assurance  Process 


Software  Verification 
Process 


Software  Validation 
Process 


Software  Review  Process 


Software  Audit  Process 


Software  Problem 
Resolution  Process 


Adapted  from  WG7  N1111; 

Adapted  15288  Outcome/s 

One  or  more 

Blended  12207  &  15288 

One  or  more 

12207-based  Outcome/s 
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Activities,  Tasks 

12207  Outcomes 

Activities  and  Tasks 

15288  Outcomes 

Activities,  Tasks 

Building  12207:2008  -  System 


Context 


12207  Operation  and  Maintenance  Processes  complete  the  System  Context 


Agreement 


Acquisition  Process 

Supply  Process 

Organizational 

Project-Enabling 

Life  Cycle  Model 
Management  Process 


Infrastructure 
Management  Process 


Project  Portfolio 
Management  Process 


Human  Resource 
Management  Process 


Quality  Management 
Process 


Project 


Project  Planning  Process 


Project  Assessment  and 
Control  Process 


Decision  Management 
Process 


Risk  Management  Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


Technical 


Stakeholder  Reqmts 
Definition  Process 

System  Requirements 
Analysis  Process 

System  Architectural 
Design  Process 

Implementation  Process 

System  Integration 
Process 

System  Qualification 
Testing  Process 

Software  Installation 
Process 

Software  Acceptance 
Support  Process 

Software  Operation 
Process 

Software  Maintenance 
Process 

Software  Disposal 
Process 


SW  Implementation 


Software  Implementation 
Process 

Software  Requirements 
Analysis  Process 

i 

Software  Architectural 
Design  Process 

Software  Detailed 
Design  Process 

Software  Construction 
Process 

Software  Integration 
Process 

i 

Software  Qualification 
Testing  Process 

SW  Reuse 


Domain  Engineering 
Process 


Reuse  Asset 
Management  Process 


Reuse  Program 
Management  Process 


SW  Support 

Software  Documentation 
Management  Process 


Software  Configuration 
Management  Process 


Software  Quality 
Assurance  Process 


Software  Verification 
Process 


Software  Validation 
Process 


Software  Review  Process 


Software  Audit  Process 


Software  Problem 
Resolution  Process 


Adapted  from  WG7  N1111; 

Adapted  15288  Outcome/s 

One  or  more 

Blended  12207  &  15288 

One  or  more 

12207-based  Outcome/s 
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Activities,  Tasks 

12207  Outcomes 

Activities  and  Tasks 

15288  Outcomes 

Activities,  Tasks 

Building  12207:2008  -  Software  Specific 


Software  Specific  Support  almost  the  same  as  12207  Supporting  Processes 
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Another  Way  of  Looking  at  It 


Revised  Content  (Viewed  from  12207) 


Revised  Standards 

■  Front  Matter 

1.  Scope 

2.  Conformance 

3.  Normative  References 

4.  Terms  and  Definitions 

5.  Application  of  this  International  Standard 

6.  System  Life  Cycle  Processes 

7.  Software  Life  Cycle  Processes  {Italicized  indicates  12207  Only} 


The  12207  Annexes  (12207  and  15288  differ  somewhat  in  format  and  content  here) 

a.  Tailoring  (Normative) 

b.  Process  Reference  Model  (Normative) 

•  1 5504-2  Conformance,  PRM  Lower  Level  Processes  for  Acquisition,  Supply,  Life  Cycle  Model  Management, 
Human  Resource  Management,  and  Software  Operation 

c.  History  and  Rationale  (Informative) 

•  History,  Process  Integration/Constructs  and  Usage,  Relationships,  Process  Definition  Sources 

d.  Process  Alignment  of  1 2207-1 5288  {Clause  6}  (Informative) 

e.  Process  Views  (Informative) 

•  Concepts,  and  Process  View  for  Usability  Example 


f.  Some  Example  Process  Descriptions  (Informative) 

g.  Relationship  to  other  IEEE  standards  (Informative) 

h.  Bibliography  (Informative) 

i.  List  of  {IEEE}  participants  (Informative) 
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Aligned  15288  and  12207  Set  Provides 


■  Coordinated  Terms  and  Definitions 

■  Integrated  Process  Structure 

■  Coordinated  Process  Sets 

Backward  compatible 

□  Usable  stand  alone  or  jointly  by  systems  and  software  teams 

□  System  Context  processes  are  nearly  identical  or  the  12207  processes 
provide  software-appropriate  specializations  of,  or  contribute  to  the 
outcomes  of,  the  corresponding  15288  processes 

□  Especially  on  Agreement  and  Project  Processes 

■  Common  Conformance/Tailoring 

■  Common  Life  Cycle  Model  and  Stage  Concepts 

■  Free  Guidance  (Annexes  and  Plan  for  TR  24748-1 ) 

Easier  Joint  Use  -  Improved  Efficiency  -  Reduced  Costs 
Common  Acquisition,  Supply  and  Management  Views 
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Towards  Full  LCP  Integration 


■  WG7  Study  Group  on  Harmonization  Integration  Strategy  Report 

SC 7  Life  Cycle  Process  Harmonization  Advisory  Group  (LCPHAG) 

■  Work  with  SWG5  across  SC 7  and  externally  for  analyses  and  recommendations 

■  Model  SC7’s  current  LCPs  and  supporting  standards 

■  Study  Process  Repository  and  Electronic  Publishing  Concepts 
Rigorous  review  of  SC7  Vocabulary  (WG22) 

Start  revision  to  15289  (Documentation)  to  reflect  aligned  set. 

■  Some  15288-12207  Integration  Considerations: 

Common  purpose  and  outcomes 

Architecture  of  the  standards 

Level  of  prescription  of  activities  and  tasks 

Life  cycle  treatments 

Application  to  services  and  operations 

Common  verification  and  validation  concepts 

Common  configuration  management  concepts 

Alignment  with  other  applicable  standards 

Rationalization  of  application  guides 


Source:  WG  7  N1 103-  Strategy  for  Integration  Study  Group  Final  Report,  22APR08 
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SC7’s  Large  Scale  Harmonization  Efforts 


SWG1 

•  Business 
Planning 

SWG5 

•  Standards 
Management 


Study  Groups,  e.g. 

•  Relationships 

•  Integration 


LCPHAG 

•  Modeling 
•Architectural 

Analysis 

•  Process 
Repository 


Overview  of  the 
SC  7  collection 


Governance  ] 

!  900  l"!s 

Quality  System 

29151 

38500 

.-GmernaocR. 

_ J 

Foundation  j 

24765 

Vocabulary 

24774 

Process  Description  < 

r— 

BOKand 

Professionalism  j 

19759 

SWEBOK 

24773 

29154 

Certification 

Product 

Characteristics 


Tools, 

Methods 


3535,5806,5807 
8631,8790, 11411,  14579 


SC7  Legacy  Standards 


12182 


14102,  14471 
15940,  18018 
23026,  29118 
24766 

Tools,  methods 
and  environment 


Need  to  Add 


10746,13235 
14750,14752 
14753,14769 
14771,15414 
15935, 19500 
19770-2,3 


15437 

15909 

19501 

19739 

8807 

24774 


14568 

15474 

15475 

15476 


Draft 


Interchange 


Version  14.2 -May  2008 


Specifications 


Modeling 

Note:  Italicized  number  =  standards  or  TRs  under  development 


Source:  Adapted  from  ISQ/IEC  JTC1/SC7  SWG5  V14.2  briefing  material,  May  2008 


Solid  Yellow  =  Aligned  or  in  process; 
Gradient  Yellow  =  Planned  for  alignment 


Te 
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Harmonization  Across  Collections 


The  State  of  Harmonization  ...  Today 


Topic 

Status 

Ferraras 

Taon-fiotagy  i  Concept* 

V»Ko*t 

Shared  BCf  joint  vc«abjUry  project  pc  tart  3  cerbficetior  f-amework 

Quality  nv»r»ag*rri*rtt 

Yellow 

'EEE  •»  adoptmg  ISOilEC  *0002  approach 

"eating 

Oranga^ 

Both  IEEE  and  SSI  *1  harmorvze  with  SC7  pioc*it« 

A'et'itoctJ’*  a«scn»t»o« 

SCT  adopted  IEEE  standard  ana  w»i  hamvona#  with  processes 

Product  quality 

•allow  * 

tSOdEC  12119  wa*  tev>Md  as 29051.  f  tE  *.!  wt omw  •►  stancem 

Ufa  eye'# 

5pl»-i»  engineering 

• 

Shanao  SE  orocess  standard  harmonisation  wth  other  lC  process#*  underway 

5W  ma>rna<ianc* 

Project  to  merge  IEEE  and  ISO  standards  <s  completed 

Measurement 

•allow  f 

EEE  *t  adopt  19939  attar  its  aonant  'anvan  Soma  eela  s  iamaut 

Risk  management 

SC7  adopted  IEEE  standard  and  is  now  artand.ng  It  to  0>#  system*  level 

Preset  management 

•allow  * 

Project  is  mengmg  the  tncompatibte  sUnoartti. 

Ve-iAcat cr>  ane  vacation 

|  Fundamental/  different  aoproache*.  Good  intentions,  tut  no  action  yet 

Configurator  management 

•  allow 

SC"  withdraw  its  standard  systems  issues  rema-n.  IEEE  •»  aoovt  to  r*>  -us 

SW  preem  assessment 

•allow* 

htarmonaat  on  wtn  LC  p  rocess  standard*  is  underway 

O'a-g*  if 

Jcw-t  project  has  oeen  approved  nirs-uo  or  »»*<•)•  .»  L.ng  prep  v«  t 

SW  bft  cycle  data 

•  allow  * 

•EEE  <*  adco&ng  15289  to  ropf-ace  12207.1 

User  decumensasen 

VaHow  * 

IEEE  1053  has  been  incorporated  Into  2C5M  IEEE  n  *  *o*f  1 

CASE  tocis 

•'allow 

M  nor  rvsompaapiitas 

Notation* 

Harmless 

Ost  ncl  standards  for  district  rotations 

Internal 

5w 

Shared  standard 

IT  Smcrt  Management.  Governa-ioe 

Yellow 

'EEE  w  f  eovJt  2DCO?  Slat  -Janas 

Specialty  Erg  nearing  (Safety.  Security  i 

Orange  + 

Uni# ta«M  approaches  a-  he  addressed  in  van  or  ceowanekon  revision  of  ISOM 

Otner* 

fallow 

Many  unreateo  standard* 

The  State  of  Harmonization  in  1995 


Topic 

Status 

Remarks 

Terminology  &  Concepts 

Different  vocabulary  standards 

Quality  management 

Orange 

ISO:  Driven  down  from  ISO  9001.  IEEE:  traditional  QA  approach. 

Testing 

Orange 

IEEE  standards  unrelated  to  SC7  processes 

Architecture  description 

Harmless 

SC7  didn't  have  architecture  standards. 

Product  quality 

Unrelated  standards 

Life  cycle  processes 

Incompatible  standards 

Systems  engineering  process 

Unrelated  standards 

SW  maintenance 

Incompatible  standards 

Measurement 

Yellow 

Unrelated  standards 

Risk  management 

Harmless 

No  standards  at  all 

Project  management 

Incompatible  standards 

Verification  and  validation 

Fundamentally  different  approaches:  minor  incompatibilities  in  details 

Configuration  management 

Incompatible  standards 

SW  process  assessment 

Yellow 

Nothing  in  IEEE.  ISO  process  assessment  incompatible  with  ISO  LC. 

Requirements  engineering 

Orange 

IEEE  standards  unrelated  to  SC7  processes 

SW  life  cycle  data 

Incompatible  standards 

User  documentation 

Incompatible  standards 

CASE  tools 

Yellow 

Minor  incompatibilities 

Notations 

Harmless 

Distinct  standards  for  distinct  notations 

Internet 

Harmless 

No  standards 

IT  Services.  Management.  Governance 

Harmless 

No  standards 

Specialty  Engineering  (Safety.  Security) 

Orange 

Unrelated  approaches 

Others 

Yellow 

Many  unrelated  standards 

IEEE  CS  May  2008 
Status  Report  to  SC7 

Stoplight  charts 
show  marked 
improvement 
between  the 
IEEE  and  SC7 
Standards 
Collections 
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Harmonization  Benefits  Summary 


Alignment 

■  Achieves  short  term  objectives 

■  Maintains  backward  compatibility 

■  Starts  disparate  users  towards  goal 

Integration 

■  Tackles  the  ‘religious’  issues 

□  Technical  and  Political 

■  Achieves  long  term  goals  in  a  set 

Large  Scale  Harmonization 

■  Solves  big  picture  issues  within  and 
across  SDOs 


Each  Level  Brings  You 

■  Easier  process  definition  and 
implementation 

■  Better  team  communication 
and  integration 

■  Improved  performance  at 
lower  cost 

■  Increased  benefit  and 
usefulness  of  implementing 
these  standards  in  your 
organization 


Eases  Your  Integration,  Management,  and  Acquisition  Burden 
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For  More  Information  Contact 


■  Teresa  ‘Terry’  Doran 
TECHSOFT 

31  West  Garden  Street,  Suite  100 
Pensacola,  FL  32502-5685 
Internet:  www.techsoft.com 


NY  Office  Tel:  1  631-266-2191 
Email: 

■  ISO/IEC/IEEE  12207  Project  Editor 

■  15288-12207-24748  Editorial  Team  Member 

■  IEEE  Std  1 220™-2005  Project  Editor  (aka  ISO/IEC  26702:2007) 

■  ISO/IEC  JTC1/SC7  Life  Cycle  Process  Advisory  Group  Chair 
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Abbreviations  - 1 


ANSI 

CMMI 

CMU 

IEC 

IEEE 

IEEE  CS 

INCOSE 

ISO 

IT 

JTC1 

LCP 

NWIP 

OPA 

OPD 

SC 

SG 


-American  National  Standards  Institute 

-  Capability  Maturity  Model  Integration 

-  Carnegie  Mellon  University 

-  International  Electrotechnical  Commission 

-  Institute  of  Electrical  and  Electronics  Engineers 

-  IEEE  Computer  Society 

-  International  Council  on  Systems  Engineering 

-  International  Organization  for  Standardization 

-  Information  Technology 

-  ISO/IEC  Joint  Technical  Committee  1:  Information  Technology 

-  life  cycle  process 

-  new  work  item  proposal 

-  organizational  process  assets 

-  organizational  process  definition 

-  subcommittee 

-  study  group 


SEIfianncr 

CHSQFT 


Technical  Software  Services,  Ire. 


TSDoran-NDIA-SE  23OCT08  vl.O 


43 


Abbreviations  -  2 


SC7  -  ISO/IEC  JTC1  SC  7:  Software  and  Systems  Engineering 

SE  -  systems  engineering 

SEI  -  Software  Engineering  Institute  (at  CMU) 

S2ESC  -  Software  and  Systems  Engineering  Standards  Committee  (IEEE  CS) 

SEP  -  SE  process 

SWE  -  software  engineering 

SWG  -  special  WG 

WG  -  working  group 

WG7  -  ISO/IEC  JTC1  SC7  WG  7:  Life  Cycle  Management 
VSE  -  very  small  enterprise 
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References  - 1 


For  ISO  and  ISO/IEC  Standards  (Current  and  Withdrawn): 

http://www.iso.orct/iso/iso  cataloaue.htm 

1)  ISO  9001 :2005,  Quality  management  systems  —  Requirements 

2)  ISO/IEC  1 2207:2008,  Systems  and  software  engineering  — 
Software  life  cycle  processes 

3)  ISO/IEC  1 5288:2008,  Systems  and  software  engineering  — 
System  life  cycle  processes 

For  ISO/IEC  documents  and  in-process  standards  and 
technical  reports  (TRs):  hfto://www./tcf-sc7.org/ 

4)  SC7  N4143:  ISO/IEC  DTR  24748.2:2009,  Systems  and  software 
engineering  —  Guide  for  life  cycle  management 
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References  -  2 


For  IEEE  Standards: 

http://www.ieee.org/web/standards/home/index.html 

IEEE  Std  1220™-2005,  IEEE  Standard  for  Application  and 
Management  of  the  Systems  Engineering  Process 

Or  related  information: 

http://standards.computer.org/s2esc/ 

IEEE  CS  Software  and  Systems  Engineering  Standards  Committee 
-  for  on-going  SE/SW  standards  activities 

h ttoJ/pascai.  computer  org/sev  displa  v/index.  action 

SEVOCAB:  An  IEEE  CS  and  ISO/IEC  JTC  1/SC7  project,  SEVOCAB 
includes  definitions  from  international  standards;  This  database  is 
issued  periodically  as  a  formal,  published  International  Standard 
(ISO/IEC  24765)  reflecting  a  "snapshot"  of  the  database. 
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SLR/ICE  a 

ENGINEERING  COMPANY 


Near-field  RCS  and  Fuze  Modeling: 
Assessment  and  Strategy 


NDIA  Systems  Engineering  Conference 

Oct  22,  2008 


David  H.  Hall,  Dorothy  L.  Saitz,  Dr.  David  L.  Burdick 
SURVICE  Engineering  Company 
Ridgecrest,  CA 


Objectives 


*  In  an  encounter  between  an  aircraft  and  a  missile,  fuze  function  is 
one  of  the  most  important  endgame  elements  in  determining  the 
probability  of  kill  (Pk) 

*  In  recent  years,  proximity  fuze  modeling  and  the  required  near¬ 
field  RCS  modeling  do  not  appear  to  have  received  adequate 
attention 

*  This  effort  is  investigating  the  state-of-the-art  of  proximity  fuze 
modeling 

•  Our  goal  is  to  help  determine  the  need  for  resurrecting  and 
improving  this  capability 

•  We  are  actively  seeking  information  on  who’s  doing  what  with 
which  kinds  of  models 

•  We’re  interested  in  all  kinds  of  fuzes: 


•  IR 


•  RF 


•  Active  Optical 


•  Guidance  Integrated 
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Applications 


•  SYSTEM  LETHALITY 


SLR/ICE 
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V 

Typical  Surface-to- 

Engagement 


ACQUISITION/ 

DETECTION/TRACK 


LAUNCH 
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Endgame  Models 


OUTPUT  FROM 

FLYOUT 


•MISSILE  VEL  COMPONENTS 
•TARGET  VEL  COMPONENTS 


Z 


•  What  happens  after  the  last  missile  guidance  time-constant  before  intercept 

•  Everything  is  assumed  to  be  a  straight  line 

•  Acceleration  is  assumed  to  have  little  or  no  effect  during  endgame 

•  Calculate  events  along  the  relative  missile-target  velocity  vector  (Vmt) 

•  Fuze  Declaration  Position 

•  Warhead  Burst  Point 

•  Impact  with  Target  (if  direct  hit) 
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rFuze  Model  Within  the  Endgame 


Inputs: 

Encounter 

Geometry 


No  Target  Detection 


Fuze 


Endgame  Integration  Model 

•  Warhead  Model 

•  Target  Vulnerability  Model 


Target 
Detection 
(on  Vmt) 

1 

r 

Warl 
Burst 
(on ' 

lead 

Point 

i/mt) 
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V  Fuze  Model  Elements 


SIGNAL 

PROCESSING 
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Modeling  a  Proximity  Fuze 


FUZE 

•TRANSMIT 

•RECEIVE 

•LOGIC 


ANTENNA 


TARGET 

MODEL 


ANTENNA 


MORE  DIFFICULT  LESS  DIFFICULT  MOST  DIFFICULT 


RELATIVE  MODELING  DIFFICULTY 


SIR/ICE  a 

ENGINEERING  COMPANY  + 


Example  Near  Field  Signature  Methodology: 
Geometrical  Theory  of  Diffraction  (GTD) 
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r  Missile  Engagement  Simulation  Arena  (MESA) 


Realistic  Encounter  Simulations  Provide: 


•  Fuze  Performance  (Pd) 

•  Warhead  Burst  Point 

•  Countermeasures  Effects 

•  Overall  Missile  Performance 

•  Effectiveness  Analysis  Support 

•  M&S  Validation  Data 


•  Unique  China  Lake  Facility  for  Evaluation  of  Missile  Proximity  Fuzes  Against  Full  Scale  Targets 

•  Effects  of  Near  Field  Signatures  (Aircraft  or  Missile)  on  Threat  Missile  Fuze  Performance 
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Example  Measurements  vs. 
GTD  Model  “Crayola”  Target 
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What  Drives  Pk  the  Most? 

How  Good  Does  the  Fuze  Model  Need  to  Be? 


•  Sensitivity  Analysis  Can  Support  the  answers: 


•  Determine  Effect  on  Pk  Caused  by  Errors  in  Inputs  to  the 
Endgame 

•  Compare  results  to  Pk  accuracy  requirements  for  specific 
applications 

•  Example:  Net  Reduction  in  Lethality  (NRL)  for  ECM 

NRL  =  1  -  Pk<wet> 

Pk(dry) 
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Endgame  Parameters  Affecting  Pk 

•  Primary  parameters 

•  Intercept  geometry  parameters 

»  Miss  distance,  direction 
»  Vm,  Vt 

»  Approach  angles 
»  Angles  of  attack 

•  Fuze  declaration  position  [on  Vmt] 

•  Target  Vulnerability 

*  Secondary  parameters 

•  Fuze  parameters:  detection  thresholds,  etc. 

•  Warhead  parameters:  ejection  angle,  etc. 

•  Fault  trees:  redundancies,  etc. 
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Example  P(K)  Sensitivity 

to  Fuze 

Detection  Position 


FUZE  pa  NTT  RANGE 
(DELTA) 

Figure  1-2.  P(K)  Profile  Along  Vmt 
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IN  ORDER  TO  ACHIEVE  A  SPECIFIED  ACCURACY 
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0  5  10  15  20 

INTERVAL  (FEET)  ON  VMT 

Figure  1-3.  Interval  in  Which  Fuzing  Must  Occur  To 
Achieve  A  specified  P(K)  Accuracy 


STICK-CONE  INTERVAL  FREQUENCY 
ADDS 


Figure  I-3A.  Interval  in  VUiich  Fuzing  Must  Occur  (on  Vmt) 
To  Achieve  a  Specified  R(K/F)  Accuracy 
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Sensitivity  Analysis  Results 

Primary  Drivers  of  Pk  (in  order): 

1.  Fuzing  (Burst  Position) 

2.  Miss  Distance 

3.  Az 

4.  El 

5.  Yaw 

6.  Pitch 

Relative  importance  depends  on  specific  intercept 
conditions,  type  of  missile  and  type  of  target 

It  Is  Impossible  to  Know  the  Validity  of  Simulated  Pk 
Without  Knowing  the  Validity  of  the  Fuze  Model 

•  Errors  in  fuzing  prediction  can  change  the  predicted 
Pk  from  zero  to  one  or  vice  versa 
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V 

Modeling  Fuze  Performance 


•  Models  of  proximity  fuzes  require  simulation  of  near  field 
signatures  as  well  as  fuze  system  (sensors,  processing) 

•  Some  options  include: 

»  Simple  geometric  model  (stick-cone  model) 

»  “Advanced  Fuze  Model”  in  models  like  ESAMS,  SHAZAM 
»  Near  field  signature  models  (GTD,  PTD) 

•  Risk  Areas: 

•  Some  elements  of  threat  fuzes  not  well  understood 

»  Burst  Control  Logic 
»  Detection  algorithms 

•  Stick-cone  model  does  not  well  represent  threat  fuze  characteristics 

•  Models  like  ESAMS  advanced  fuze  model  have  little  or  no  usage 
history  nor  any  documented  V&V 

•  GTD,  PTD  signature  models  require  development  for  use  with  fuze 
models 
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Project  Objectives 


•  ID  current  approaches  to  Proximity  Fuze  modeling 

•  Government  and  Industry 

•  Document  the  “State-of-the-Art” 

•  Determine/Examine  needs  for  improvement 

•  Methodology 

•  Data 

•  Verification  and/or  Validation 

•  Develop  a  strategy  for  improvement 

•  Develop  a  plan  for  filling  methodology,  data  &  V&V  gaps 

•  ID  potential  funding  sources 

We  are  actively  seeking  information  on  the  current  status  of  fuze 
modeling  in  Government  and  Industry  (and  in  other  countries) 
Please  let  us  know  if  you  have  any  information! 
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The  Open  Architecture  Imperative 


The  Navy  must  build  a  fleet  where  our  systems 


...  are  modular,  interoperable,  and  affordable  to  upgrade 
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To  accomplish  this,  ASN  (RD&A)  in  2003  commissioned  a  Red 
Team  to  assess  the  Navy’s  plan  to  adopt  Open  Architecture 

The  Red  Team  Made  13  Recommendations  to  leadership: 


1.  Develop  and  promulgate  a  clear  Navy  policy 

2.  Develop  a  Navy-wide  business  strategy  to  support  OA  goals 

3.  Redirect  the  OA  implementation  by  defining  architectures  for  domains 
based  on  their  unique  needs 

4.  Assign  one  PEO  to  be  accountable  for  managing  OA  in  each  domain 

5.  Investigate  alternate  strategies  for  budgeting  and  contracting  for  ships 
and  their  combat  systems  to  maximize  benefits  of  open  architectures 

6.  Evaluate  DDX,  AEGIS,  LCS,  and  CVN/large  deck  L-ships  combat 
system  requirements  and  analyze  architecture/cost  trades  to  exploit  a 
common  architecture  for  surface  ship  command  and  decision  systems 

7.  Review  all  applicable  programs  to  determine  how  OA  is  actually  being 
implemented  and  what  changes  in  the  program  of  record  are  required 
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Red  Team  Recommendations  (continued) 

8.  Reaffirm  the  role  of  PEO  IWS  in  the  Navy-wide  OA  Initiative 

9.  Modify  and  enforce  the  OA  architecture  definition  and  standards 
selection  processes  within  and  across  communities 

10.  Implement  and  sustain  a  proactive  education  and  information 
exchange  program  across  the  Industrial  and  Government  communities 

11.  Modify  testing  and  certification  processes  to  exploit  OA 

12.  Regarding  JTM  and  its  development  by  JSSEO: 

□  Determine  whether  the  technical  approach  and  the  transition 
strategy  to  Navy  programs  is  appropriately  risked 

□  Determine  whether  the  Navy  programs  have  sufficient, 
coordinated  off-ramps 

13.  Consider  using  the  basic  framework  of  these  recommendations  for 
Navy  OA  to  address  Joint  interoperability  and  network  centric  warfare 
requirements 

The  Red  Team  included  several  technical  recommendations 
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These  recommendations  acknowledge  that  many  pieces  of 
the  acquisition  puzzle  are  required  to  become  “truly  open” 

Open  Architecture 

The  confluence  of  business  and  technical  practices  yielding  modular, 
interoperable  systems  that  adhere  to  open  standards  with  published 

interfaces. 
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So,  leadership  mandated  Open  Architecture  implementation 
across  the  Naval  Enterprise  and  provided  some  guidance 


1  Aug  2004  ASN  RDA 
mandates  open  architecture 


MEMORANDUM  FOR  DISTRIBUTION 


SI  Ntvti  Open  ArcblWoure  Scope  *ml  Rc^xxiuwlltha 
End:  I IJ  Open  AnJiUetlurc  Enccqirivc  Team  Orpunzalian 


Ore  purpoic  uT  this  areaiuraaduci  n  la  amplify  ml  expand  upon  Die  pnU-y,  pnidance  mi 
direction  nece«wy  for  ine  taacextfel  implcmenurinn  of  the  V«vy’«  Ope*  Alrtulcriwe  (OA I 
Slralagy  This xlntcgy ft cweiui*  matey erabler and  |»IW*of  DoD't foctit on  faun 
atchiiectoret  and  cvolunonaiy  acquisition  1>>DD  SOHO  I  dated  12  May  2003  umi-i 
‘  AcgaKitfnn  pmgranu  -lull  be  managed  ihinijtk  II  «  eppbcauon  of  a  lyslnna  engineering 
approach  rhw  opnnutrs  WOI  sycretns  pcffireniani*  and  laiulmizc total  uwnen&ip  cosu  A 
modular,  open  sysrins  approach  dial]  l«  employ*. I  where  legible."  Thu  mandate  lu  utilize 
open  tyflcma  wi+ilroiiuiti  in  order  lu  mpmlly  field  alTonlable.  inusopcraKt  lyflemx  is 


In  lighl  at  Jib.  I  miiraud  an  effon  la  nszahluh  open  nnrliiuenue  prinmptes ill ihe  hatn 
for  all  scar  fighting  syuztro  develnpiiml  and  nmuilenance  during  ihe  In  OdrAa-.i  21)03  Navy 
Open  Arduiccinre  Exetulrm  Coenmiuee  (EXfOMMi  The  plan  was  nngfivdly  timed  on  the 
fcnudaltaa  or  a  single  Navy  (1A  Technical  A/cbiiccrure, »  wngle  Navy  <3  A  Fimalniuii 
Arehfwaure  and  armoncling  bed  of  breed  telocrinn  for  common  servlcet  Alter  reviewing  OA 
pnrgyew  lo  dale  and  the  irtulta  of  Ihe  OUST)  IATAXi  Tn- Service  Independent  AMrtttwm 
■land*  Ihe  ircood  Open  AJCArtceuxc  EACOMM  2  time  JftH  I  have  cooellnlcd  ilial 
ntudincauc*  lo  this  plan  a  ncccvsurv  Tlie  apoioach  to  develop  a  magic  Saw  wide  Open 
Aniltflcetuic  will  le  ntudllxod  lu  accuiBl  tur  Sutto  s,  Ala.  Suboutrim.  CAI.  nl  Space*  duiaam 


Effective  immediately,  PEO IWS  i»  atHjurd  overall  responsibility  and  aulhnnty  lor 
dneoing  the  Navy's  OA  Fnxsrprvre  tffun  An  OA  Pmeipnse  Team  shall  be  chanenrd  and  led  hr 
PEO  IWS  The  Team  shall  be  comprised  of  OA  donstsn  leads,  VSS,  OPNAV.  and  SVSTOM 
represenlaiises.  who  will  collectively  oversee  the  i.'evelnpraem  and  implemcnlasion  of  Ihe 
peooeates  hnuiiCM  dmtegrev  and  technical  whumc*  which  support  .  tor*  Emerprue 


Naval  OA  Policy 


Dec  2005  OPNAV  issues  OA 
Requirements  letter 
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©DEPARTMENT  Of  THE  NAVY 

ot'ts.t  or  Twr  pny p  or  naval  operation 

.'Jun  RAVt  PENTAGON 
WAStWfCION  oc  yoaspiopp 

9010 

Ser  N6N7/  SU916276 
23  Dac  OS 

From:  Deputy  Chief  of  Naval  Operations  (Warfare  Requirements 
and  Programs)  (N6/N7) 

Subj :  REQUIREMENT  FOR  OPEN  ARCHITECTURE  (OA)  IMPLEMENTATION 

Ref:  (a)  ASN (RDA)  Memorandum  on  Naval  Open  Architecture  Scope 

and  Responsibilities  dated  05  August  04 

Enel:  (I)  OA  Enterprise  Team 

1.  Purpose  This  letter  establishes  the  requirement  to 
implement  Open  Architecture  (OA)  principles  across  the  Navy 
Enterprise.  To  deliver  timely,  affordable,  interoperable 
warfighting  capability  to  the  fleet,  made  sustainable  by  the 
flexible  integration  of  emerging  capabilities,  we  must 
incorporate  OA  processes  and  business  practices  now. 

2.  Background  Warfare  systems  include  hardware,  software  and 
people.  Human  factors,  (i.e.  such  as  training,  education  and 
doctrine)  factor  heavily  in  warfighting  effectiveness.  Naval  OA 
transformation  must  match  the  rapid  evolution  in  conmercial  and 
military  technology.  Not  only  must  we  shorten  the  kill  chain 
across  the  family  of  systems;  we  must  also  shorten  the  time  and 
cost  it  takes  to  deliver  capability  improvements.  Our  current 
process  takes  nearly  a  decade,  costs  hundreds  of  millions  of 
dollars  and  delivers  products  that  are  commercially  obsolete  and 
have  only  incremental  improvements  in  warfighting  capability. 

That  is  not  good  enough,  and  must  change  in  POM 08 .  Acquisition 
processes  and  business  practices  must  transition  now  in  order  to 
support  POM  08  and  implement  agile  changes  that  support  rapidly 
evolving  requirements. 

OA  Principles  include: 

*•  Modular  design  and  design  disclosure  to  permit 
evolutionary  design,  technology  insertion,  competitive 
innovation,  and  alternative  competitive  approaches  from  multiple 
qualified  sources. 


Naval  OA  Requirements 


OA  CORE  PRINCIPLES 

Modular  design  and  design 
disclosure 


Reusable  application 
software 


Interoperable  joint 
warfighting  applications  and 
secure  information  exchange 


Life  cycle  affordability 


Encouraging  competition 
and  collaboration 
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From  this  guidance,  the  OA  Enterprise  Team  (OAET)  developed  a  Naval 
OA  Strategy  that  includes  goals,  objectives,  practices,  and  tools  ... 


OA  STRATEGY 
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Naval  OA  Strategy 


© 


~'FrvbsNg  the  btvgni  challenge  I  fle-r  n  to  get  the  ship  buHtfing  ktty  rtvhi  ht  gri  rhr 
fonts  tcnpebinliis  right.  Wear tsstlSl  s^ijp-s  today,  Jf>  have  cawt  daw ra,  anal 
beTseve  ars  protected  if  go  up  —  and  ipi  need  Us  rasfajn  that  proieeden  to  2  soiish'e  dtceijan 
-  .flSji  Mullsc.  CfcsafflfS^rai  Cpcro*;  16  Ctr  3XT 

Wiry  It.-.iianlus  i:  ficii  sr.ib  fc-nus  :■ 

:=d ^'iilvicz  uacci.  sararin--  ucyjzaiuau;:  a1l.Ii  il  ti.s  1534  r.i&*  c  jut: ILte  -ii  -■js.i  121 
“/Mini  ire  2  lll.j  plalroscu.  k  UTdHT  In  build  jambaE  syiJhiu  capable  ot  jjnnlstiiLT  anai 
;u»cmdc  nuiiijvn  ut:d:  i  it  suran  aur  mragjnsi  w*  bli:i  k  abJa  »  qpickiy  iuaa dura . 
iKS.24k:i»s  Tiro  ;b«  71a  ai.  Changing  s.  wiy  wa  dc  '.tsiz.:  ; :  today  1 : 
impsTarivn  if  a  a  an-  :&  gam  cu  idfsd  Ei-adb Jir;  tKjuzad  so  ripiati*  on  k™ 
laokmijain:  iz  i  dalirsf  ±s  nsLl  tap afcdidias. 

Kits!  Opon  ArtlnsKrart  (0-A) 11  m 

ttrcimuil  rinsaiy  fur  ai-gunrg  and  ouniijnmg  N nisei!  5-rcmrty  SjnmiE  a; 

:uo;rcp  sT.tli  y.tami  tbr.  adnet  usd  air.ii;  span  iyimcl  dangs  pnzcialai  is 
aztlififtjTirs-L.  Tk;  zsidili1: «  u  a  toy  auabis:  and  piGu  of  I  ha  Dapartczcud  zi 
Difanii’i  ilioD;-  6cui  es  jnuii  auiitacmaas  acd  arobuicsz^jcguiitlicz. 

DusC  Dkautrva  jjfO.l  iitad  52  Mi;  25B3  tlalai  "Atoaivsbous  pro  cnci>  .dill  be  tiil.ii;.z 
ipobcar.oD  cl  a  rj-pami  aniiOKr-is  appcn-adi  til  agnmiK;  Era-  sytiaui  pjdiecTEaitg  jlo  ai^usuT.T  tcJiJ 
awuanb:?  lusK.  A  znnadzr.  span  zyrtaM*  aypro-sas  skill  bd  ampk-ysc,  wbars  firanMa." 

Byadoptms  DAzdudpia;  tisonghixa  As  siral  nluprisn  today.  v.  4  eta  tviic  xatzilir  xSordsbln.  nii=s« 
:«nb4t  sYTiMn.1.  dragsM  in  3»!  tun  &mre  Med:  zi  sin  Saits.  Tcaia  lyicsst:  -wall  siu  be  jbh  in  laieily 
lucd.'pcsH  satarriGE  of  new  webuiaf..:  tjib  1  :to?d  :ic;;  of  !udv.:c)  puaeii.  Efnwerir,  s.  d~>  C  MG  :nro ; 
w:  muii  be  ton:*  lsa&ra  inr  jvlilol  usd  tbaugo  to  losScs  :bi.  iiKX-i.  V*  ur.:.-  idantif;,'  c:irpr±  sorwiri. 
ULt  icatoay  ]zyr  onr  ia  Kny'  s  -nsioc..  ecol  13d  cbetorBS  izz  nuctoaLlLET.E'  t3A  ouwir  'lie  Miwrpciio 

Naval  OA  Vision 

To  nhMC  J|«  CKGr!  pToobii  It  eamos  ^rasiu:  tmudiu  tauld  n  tletifiiros  rtmi  izd  daualop  Il“  ttoiiLiy 
insduk  Iko Natal  2 A  uiziac.  u  :o: 

iTtupforw  Disj  orTOUiDiPicm-  irrd nAasrt  and  sar  rammer  iL1  idunr  rnn  vuntw>jna\UB  -span 
^rCmtacjunEnnciplE  xvduroceiLOL  rkrohj^ocii  l^ntnuJ  camnuKr,  if)  order  ra  dalrti  'titrre 
■iivjrfF.fojVF  csmabitiiier  10  cornier  orrrenr  and  fours  threats. 


il  GA  LuoualjTi!  [2-:  Karai  :oussu3:t  1  seatsb* 


■  AL  jT  RMnn=i»-ar-. 

♦  i'-'ZZi  ?J0OU~  2L-1 IISII  IjZS-Z,  . 

■  C:L,i  fejEE  cin  42  J-  r.injaJ 

^rBiccKTta  -prodii^-  aai  a-tnt 

■  ^ w  ■ 

Zu  4rpr. ! :  izi  tvrth  foLzl  -iim :  r, 

■  AL-jt  icacy  12  d  Arid:cui 

1  fr-jisL  ijrauujse:  T.:gl! k'iii. 
jiropscr-  rjl:-: 

■  ic  rg±j:s  TiE 

LLU-.tIi  rfp-nmihB!  of  IIU^43 

TOOLS  TO  ASSIST 


OA  GOALS  OA  PRACTICES 


1.  Change  the  Naval  processes 
and  business  practices  to 
"utilize  open  systems 
architectures  in  order  to 
rapidly  field  affordable, 
interoperable  systems.” 

2.  Provide  OA  Systems 
Engineering  leadership  to 
field  common,  interoperable 
capabilities  more  rapidly  at 
reduced  costs 

3.  Change  the  Naval  and 
Marine  Corps  Cultures  to 
Institutionalize  OA  Principles 


Disclose  design  artifacts 
Negotiate  appropriate  data  rights 
Foster  enterprise  collaboration 
Reuse  GOTS  products 
Institute  Peer  Reviews 
Develop  new  business  models 
Incorporate  OA  in  contracts 


Publish  interfaces 
Isolate  proprietary  components 
Use  widely  adopted  standards 
Modularize  systems 

DAU  OA  Training 
Outreach 

Government  Symposia  &  Industry  Days 
NPS  Research 


OA  Contract 
Guidebook 


OA  Assessment 
Tool 


Reuse  Licensing  OA/FORCEnet 
Agreement  Experiment 


SHARE 

Repository 


OA  Training 
Module 


Industry 

Days 


OA  Website 


Benefits  of  Open  Architecture 


r 


...  and  found  that  implementing  OA  yields  many  benefits 


Reduction  in 
Time  to  Field 

■  Decreased  development  and  acquisition  cycle  times  to 
field  new  warfighting  capabilities 

■  Faster  integration  of  open  standards  based  systems 

Increased 

Performance 

■  Improved  operator  performance  thru  delivery  of  cutting 
edge  technologies  and  increased  bandwidth  capabilities 
from  spiral  developments  and  technology  insertions 

Improved 

In  teroperability 

■  Use  of  common  services  (e.g.  common  time  reference) 

■  Use  of  common  warfighting  applications  (e.g.  track  mgr) 

■  Use  of  published  interfaces  to  standardize  collaboration 

More  Competition 

■  Modular  architectures  enable  competition  at  the 
component  level 

■  Sharing  data  rights  allows  third  parties  to  compete 

Cost  Avoidance 

■  Cost  avoidance  from  software  reuse  and  use  of 
commodity  COTS  products  at  optimum  prices 

■  Reduced  training  and  streamlined  lifecycle  support 
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Open  Architecture  Practices 


Therefore,  the  Navy  is  changing  its  business  and 
technical  practices  to  take  advantage  of  OA’s  benefits 


Business 

Practices 


□  Disclose  design  artifacts 

□  Negotiate  appropriate  data  rights 

□  Increase  enterprise  collaboration 

□  Institute  reviews  of  solutions 

□  Develop  new  business  models 

□  Change  contracts 

□  Increase  competition 

□  Design  for  lifecycle  affordability 


Technical 

Practices 


□  Modularize  systems 

□  Publish  interfaces 

□  Isolate  proprietary  components 

□  Use  widely  adopted  standards 

□  Re-use  software  components 

□  Build  interoperable  applications 

□  Ensure  secure  data  exchange 

□  Implement  common  solutions 
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Examples  of  Open  Architecture  Implementation 


For  example,  PEO  IWS  is  building  a  modular,  common 
combat  system  architecture  ... 


Aligning  platform  combat  systems  ... 


“I  expect  us  to  compete  whenever  possible. 

Competition  provides  us  with  options  to 
seek  the  best  solution  for  the  fleet  and  the 
taxpayer.  ...  I  also  expect  us  to  foster  an 
environment  in  which  competition  can  be 
sustained  over  time.  Competition  once 
does  not  serve  our  interests.  ” 

— VADM  Paul  E.  Sullivan 


...to  one  open,  objective  architecture  ... 


...to  achieve  commonality  across  multiple 
ship  classes  where  the  business  case 
supports  it 


...  to  help  increase  competition 
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PEO  C4I  is  developing  new  business  models  ... 


Today 


C4I 


Tomorrow 


BAMS 


ECRU 


Large  acquisition  programs 
delivering  hardware  and 
software 


Integration  occurs  at  Fleet 
installations 


Limited  code  reuse 


Individual  program  DT/OT 
events 


Lack  of  platform  baselines 


Limited  competition 


Masterplan 


Integration  occurs  in  E2E 
test  lab 


Software  repository  and 
collaborative  development 
model 


Distributed  FDCE-like 
process 


Integrated  platform  C4I 
delivery 


Smaller  COI  services 
programs;  separation  of 
hardware  and  software 


Best  of  breed  process 


...  to  neck  down  and  move  towards  common  services 
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The  Importance  of  Intellectual  Property  Rights 


r 


Another  significant  cultural  change  is  that  the  Navy  now  understands 
the  importance  of  exercising  its  intellectual  property  rights 


A  key  aspect  to  implementing  OA  is  for  the  Government  to  exercise  the  intellectual 
property  (IP)  rights  it  acquires 


Under  the  Federal  Acquisition  Regulations  (FAR)  and  Defense  Federal  Acquisition 
Regulation  Supplement  (DFARS): 

□  The  Government  gets  Unlimited  Rights  in  both  Technical  Data  (TD)  and 
Computer  Software  (CS)  for  noncommercial  items  developed  exclusively  at 
the  Government’s  expense. 

□  For  noncommercial  items  developed  with  mixed  funding,  the  Government 
gets  Government  Purpose  Rights  (GPR)  in  TD  and  CS. 


If  a  contractor  asserts  more  restrictive  rights 
over  a  system/component’s  IP  and  the 
Government  fails  to  challenge  such  an 
assertion  by  exercising  its  rights,  the 
contractor  obtains  the  asserted  rights 

It  is  imperative  that  the  Government  assert 
and  exercise  the  IP  rights  it  acquires  because 
it  may  lose  those  rights  after  a  period  of  time 
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For  example,  acquiring,  asserting,  and  exercising  IP  rights  enables 
Naval  programs  to  disclose  designs  to  foster  collaboration  ... 


■  Design  artifacts  from  AEGIS,  LCS, 
DDG  1000,  SSDS,  SIAP,  IABM  are 
available  to  qualified  vendors  in 
IWS’s  SHARE  repository 


■  Project  artifacts  from  CLIP,  XCOP, 
and  NITES-Next  are  available  to 
qualified  vendors  in  the  C4I  NESI 
collaboration  site 


IWS  SHARE  REPOSITORY 


UNCLASSIFIED  ONLY! 
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In  conclusion,  over  the  four  year  span  of  this  enterprise 
transformation,  lessons  learned  have  emerged 

OA  Enterprise  Transformation  Requires... 


Clear  vision  and  strategy 

Top  leadership  support  & 
commitment 

Quick  wins  to  get  momentumj 

Enterprise  governance  & 
ownership 

Identified  Change  Agents 
Consistent  OA  Communications 
Accountability  at  all  levels 
Performance  metrics 
Fleet  driven  requirements 

Industry  /  Academia 
Involvement 

Training  /  Research 


Business  Processes 


Operational  Capability  Roadmap 
Open  /  Scalable  architectures 
Aligned  architectures 
Access  to  design  artifacts 
Published  interfaces 
Enterprise  collaboration 

Threat  /  data  driven  performance 
evaluation 

Tech  refresh  process 


Compliance  checkpoints  -  six  gate 
Consistent  assessment  approach 
Standardized  contract  language 
Knowledge  of  upcoming  contracts 
Asset  user  licensing  agreements 
Software  asset  repositories 
Changed  acquisition  bus  model 
Viable  sourcing  alternatives 
Transparency  -Third  Party  Reviews 
Streamlined  acquisition  processes 


23  October  2008 


Page  14 


UNCLASSIFIED 


Domain  Modeling 

Roadmap  to  Convergence 

Nathaniel  Homer  <-->  Steve  Topper 
22  October  2008 


"You  got  to  be  careful  if  you  don't  know 
where  you're  going,  because  you  might  not 
get  there." 

-  Yogi  Berra 


uus  LuuVcrjity 

APPLIED  PHYSICS  LABORATORY 
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Overview 

■  Introduction 

■  Conceptual  Modeling  Process  Overview 

■  Domain  Modeling  as  the  Foundation  of  the  Conceptual  Model 

■  Domain  Modeling  Application  Across  the  Project 

■  Analysis 

■  M&S,  Software  Engineering 

■  Systems  Engineering  /  Architecting 

■  Business  Processes 

■  Domain  Modeling  “Goods  and  Others” 
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Why  are  we  here? 

■  Initial  activities  on  a  modeling  and  simulation  (M&S)  project  for  a 
large,  complex,  integrated  system  attempted: 

■  To  develop  generic  DoDAF  artifacts, 

■  To  link  these  artifacts  more  closely  to  developed  models, 

■  To  provide  a  basis  for  new  M&S  development  across  a  wide 
community  of  stakeholders. 

■  Issues 

■  Legacy  tool  challenges  for  complex  systems-of-systems  analysis 
(configuration/preparation  time,  fidelity,  and  interoperability). 

■  Lack  of  standardized  foundation. 

■  Traditional  architectures  often  difficult  to  assess  using  M&S 
(lacked  underlying  referential  structure). 

■  Activities  difficult  to  accurately  plan  and  estimate. 


How  can  we  fix  it? 
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Introduction 


Problem  Domain: 

The  real-world  things  and 
concepts  related  to  the 
problem  that  the  system 
is  being  designed  to 
solve. 

Domain  Modeling: 

The  task  of  discovering 
“objects”  (classes) 
representing  things  and 
concepts,  and  the 
relationships  between 
them. 


[Rosenberg  and  Scott  2001] 
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J  HU/ APL  Dorm/  n  Mode/  i  ng 

Problem  Statement: 


Horner/  Topper  4/30 


Develop  efficient  techniques  to 
support  complex  system  analysis 


Given: 

Complex  systems,  tots  of 
components,  subsystems, 
sophisticated  behaviors, 
networks,  information 
processing,  collaboration 

Organizations  involved  in  design  <df 
development  of  these  systems 

Analysis,  requirements, 
architecture,  systems 
engineering,  software 
engineering,  testing,  operations. 


Approach: 

Understand  the  problem  Domain 
and  progress  from  there... 
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Conceptual  Modeling  Process 


"  Based  on  standard  software  and 
systems  engineering 
processes.* 

■  Translates  informal,  generalized 
information  from  disparate 
sources  into  formal  system 
models. 

"  Maintains  focus  on 
understanding  and 
standardizing  the  problem 
space  before  moving  on  to  the 
solution. 

■  Allows  iteration  and  feedback 
until  it’s  “right.” 

■  Produces  documentation 
allowing  traceability  throughout 
the  process. 


Class  Diagrams 


X 

e- 

- ] 

L 

i 

1 

ya 

yb  1 1  yc  i 

State  Diagram 

wf  class  mappings 


*  Though  significantly  changed,  this  conceptual  modeling  process  is  informed  by  ICONIX,  a 
software  engineering  process  falling  between  RUP  and  XP  with  respect  to  rigor  and  flexibility. 
ICONIX  is  documented  in  Rosenberg  and  Stephens  [2007]. 
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Conceptual  Modeling  Importance 
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“Conceptual  modeling  is  almost  certainly  the  most 
important  aspect  of  the  simulation  modeling  process .  . 
.  .  A  well-designed  model  significantly  enhances  the 
possibility  that  a  simulation  will  meet  its  objectives 
within  the  required  time-scale.  What  sets  truly 
successful  modelers  apart  is  their  effectiveness  in 
conceptual  modeling.  ” 

[Robinson  2004] 


The  first,  crucial  step  in  conceptual 
modeling  is  Domain  Modeling. 
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Domain  Model 

What  it  is:  A  “10,000-foot  view,”  a  live  “project  glossary,”  a  simplified  class 
diagram. 


How  to  do  it: 

■  Create  list  of  candidate  domain  entities  by  extracting  nouns  from  input 
documents. 

■  Review  list,  standardizing  and  defining  terms. 

■  Deploy  entities  in  a  simplified  class  diagram  (no  attributes  or  operations) 
and  draw  important  relationships  (generalization,  composition/aggregation). 

■  Iterate  as  needed  with  all  stakeholder  groups  and  revisit  throughout  the 
project 

What  it  is  for: 

■  Answers  the  question,  “What  makes  up  the  system  and  its  environment?” 

■  Defines  the  scope  of  the  project,  standardizes  terms. 

■  Provides  foundation  for  static  structural  model. 
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Domain  Model  Input 

What  it  is:  Known  information  about  the  system  and  its  environment. 

How  to  do  it: 

■  Informal  requirements  descriptions  and  mission  descriptions. 

■  NOT  detailed,  formal  system  requirements. 

■  Generalized  statements  about  system  and  what  it  does. 

■  CONOPS. 

■  Existing  documentation. 

■  Stakeholder  brainstorming  sessions. 

What  it  is  for:  Nouns  extracted  from  these  documents  form  a  list  of  candidate 
domain  entities. 
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Domain  Model  (Example) 

■  High  Level  Domain  covers  environment, 
mission  and  systems-of-systems 
representations 

■  Expands  to  increasingly  detailed  system 

representations  g. 
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Why  Use  Domain  Modeling? 

■  Standardize  and  define  the  problem  space. 

■  Use  as  a  project  glossary/naming  convention. 

■  Focus  on  real-world  (problem  domain)  objects. 

■  Document  domain  structure. 

■  Organize  around  key  problem  domain  factors. 

■  Encapsulate  (sub)  systems. 

■  Simplify  and/or  standardize  interfaces. 

■  Identify  systems  and  their  interrelationships. 

■  Enable  analysis  of  the  concepts. 

■  Provide  critical  foundation  for  follow-on  conceptual  modeling 
artifacts  (e.g.,  use  cases,  activity  models,  state  diagrams,  M&S 
software  design,  etc.). 


Complex  systems-of-systems  require  a  design  approach  that 
formalizes  the  mapping  between  behaviors  and  entities  and  remains 

flexible  and  resilient  to  change . 
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Domain  Modeling  and  Other  Project  Tasks 

■  The  domain  model  is  critical  to  the  conceptual  modeling 
tasks,  through  which  it  has  important  application  across 
analysis  and  development  projects: 


■  Research,  Development,  and  Analysis 

■  M&S,  Software  Engineering 

■  Systems  Engineering,  Architecture 

■  Business  Processes,  Project  Management 
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Why  It  Matters:  Research,  Development  &  Analysis 

■  Establishes  framework  for  factor  identification  and  selection 
including: 

■  Structure:  defines  systems  and  capabilities. 

■  Behavior:  defines  functional  processes. 

■  Defines  the  domain  entities  each  group  must  focus  on  to  achieve 
their  objectives. 

■  There  will  be  overlap  identified  -  requiring  coordination. 

■  Provides  the  terminology  and  factors  for  development  of: 

■  Tests  and  experiments  including  specification  of  alternatives  and 
trades,  and  scenario  development  requirements. 

■  System  functions  which  emerge  from  domain  entities:  methods, 
attributes,  and  interfaces. 

■  Supports  analysis  at  different  levels  of  abstraction/fidelity  without 
changing  the  underlying  model/architecture. 
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Analysis  Example 
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Results  provide  assessment  of  the  efficacy  of  the  system 
alternatives  and  the  sensitivity  of  the  factors  on  one  another. 


Targets 
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Collection  Capability 
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Analysis  factors  are  selected  using 
domain  entities  and  derived  artifacts. 

Selection  is  independent  of  simulation 
tool. 


Simulation 
implementation 
is  defined  by  the 
class  structure 
based  on  the 
domain  model. 
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Why  It  Matters:  Model  and  Simulation,  S/W  Eng. 

■  Helps  identify  where  M&S  software  should  be  developed. 

■  Represents  the  top  level  classes  and  associations  for  M&S  design. 

■  Forms  a  foundation  for  software  design  model  (UML). 

■  Models  are  derived,  developed,  or  specified  from  the  domain- 
level  superclasses. 


■  Enables  assessment  of  complex  network-centric  issues  via 
reusability,  extensibility,  and  re-configurability  of  models. 


■  Identifies  M&S  needs/requirements  for  potential  assignment  to 
available  tools  (including  legacy  simulations). 

■  i.e.,  once  a  simulation  need  is  identified,  existing  tools  can  be 
evaluated  against  it. 
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M&S  Example 


INFORMATION 

SOURCE  TRANSMITTER 


RECEIVER  DESTINATION 


SIFIED 


NOISE 

SOURCE 

_ 


■  Domain  entity  becomes  class  for  model  implementation. 

"  Model  parameters  used  to  compose  system  representation. 

■  Domain  artifacts  provide  basis  for  evaluation  of  existing  simulations. 
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Why  It  Matters:  Systems  Engineering 

■  Tracks  overall  system-of-systems  development  and  interactions. 

■  Provides  insight  into  the  system/subsystem  alternatives. 

■  Useful  as  a  foundation  for  system  architectures. 

■  Supports  requirements  development/refinement. 

■  Identifies  redundant  or  superfluous  systems/processes. 

■  Simplifies  design. 

■  Identifies  capability  shortfalls. 

■  Identifies  program  risks: 

■  Technical  readiness, 

■  Interoperability  challenges, 

■  Critical  technologies. 

■  Stored  in  a  database,  which  can  be  linked  to  other  SE  products. 
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Systems  Engineering  Example 


Requirement:  The  system  will  engage 
advanced  air-to-air  and  surface-to-; 
threats  based  on  the  rules  of 
engagement. 


- 1 

Engagement  System 

1 _ 

Inform ati  on  System 


Program  Database: 

Requirements,  domain 
model  and  other  artifacts, 
MS&A  information,  project 
management  info,  etc. 


Artificial  Features 
Systerrrof Systems 

System 


I 

I 

I 


Sensor  System 


High-level  diagram  showing  the 
functional  system  packages  that 
compose  a  Unit  (system -of- systems). 

This  structure  also  applies  to  the 
composition  of  any  unit  that 
appears  in  the  environment. 


Communication  System  | 


System-of-Sy  stems  :SoS 
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Why  It  Matters:  Business  Processes 

■  Identifies: 

■  Areas  of  responsibility  for  different  stakeholders. 

■  Maps  to  project  Work  Breakdown  Structure. 

■  Shortfalls  in  coverage/investments. 

■  Return  on  investment  and  related  tech  maturity  of  individual 
systems. 

■  Risks  to  the  overall  goals  of  the  program. 

■  How  is  this  done? 

■  Each  domain  entity  is  related  to  activities  supporting 
development  of  applications,  data  or  products  needed  to 
accomplish  objectives  and  goals. 

■  Represents  a  unified  simulation-based  acquisition  process  with  all 
components  interconnected  via  the  UML-based  architecture. 


dPL 


slide  18 


UNCLASSIFIED 


UNCLASSIFIED 


Business  Process  Example 


Task  Name 

Duration 

Start 

Finish 

-  Project  X 

56  days 

Wed  2/27/08 

Wed  5/1408 

-  Scenario  Development 

56  days 

Wed  2/27/08 

Wed  5/1408 

♦  Identify  Target  Sets 

42  days 

Wed  2/27/08 

Thu  47408 

♦  Assess  TCT  characteristics 

4  days 

Thu  4/24  08 

Tue  47908 

♦  Scenario  development 

11  days 

Wed  4*0/08 

Wed  5/1408 

♦  Intelligence  Collection  Process  Design 

1  day 

Thu  474  08 

Thu  47408 

-  Command  and  Contort  Process  Design 

1  day 

Thu  4/24.08 

Thu  47408 

Develop  Baseline  Processes/Metrics 

1  day 

Thu  4/24/08 

Thu  4/24/08 

Develop  Advanced  Processes 

1  day 

Thu  4/24/08 

Thu  4/24/08 

♦  Engagement  System  Development 

1  day 

Thu  47408 

Thu  47408 

♦  Develop  assessment  process 

1  day 

Thu  47408 

Thu  47408 

-  Model  and  Simulation  Development 

1  day 

Thu  47408 

Thu  47408 

Select  Model  Environment 

1  day 

Thu  4/24/08 

Thu  4/24/08 

^^^nTDeveiopment 

1  day 

Thu  4/24/08 

Thu  4/24/08 

System  Development 

1  day 

Thu  4/24/08 

Thu  474/08 

Sensor  Development  \ 

1  day 

Thu  4/24/08 

Thu  474/08 

Communications  Development  J 

1  day 

Thu  4/24/08 

Thu  474/08 

X.  Information  System  Development 

1  day 

Thu  4/24/08 

Thu  474/08 

^^•^Engagement  System  Devgicyanefit 

1  day 

Thu  4/24/08 

Thu  474/08 

♦  Conduct  Analysis 

1  day 

Wed  27708 

Wed  27708 

(5  Write  Report 

1  day 

Wed  27708 

Wed  27708 

■  Project’s  WBS  and  activities  based  on  domain 
entities  and  follow-on  artifacts. 

■  Enables  improved  governance. 

■  Enhances  task  estimation  and  risk  assessment. 
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Others  ,  Goods 


Domain  Modeling  Assessment 
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■  Replacement  of  legacy  applications 
(incremental  implementation) 

■  Gain  understanding  of  current  capabilities, 
analyze  costs,  compare  w/  proposed 
replacement  systems 

■  Make  future  programs  more  efficient 

■  Better  risk  management 

■  Potential  for  program-wide  database  or 
knowledge  management  system 


■  Reuse  across  portfolio 

■  Common  foundation/linkages  for  program 
tasks  (s/w  and  system  engineering, 
analysis,  business  processes) 

CONVERGENCE 

■  Standardization 

■  Greater  accessibility  to  stakeholders 

■  Lasting  documentation  (domain  longevity) 

■  Tool/simulation/code  agnostic 


■  Up-front  costs  ■  Inertia  of  DoD  acquisition  practices 

■  Understanding  new  tools,  language,  ■  Cultural  resistance 

processes 

■  Personnel  and  skillset  availability 
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Summary  --  Domain  Modeling: 

■  Is  fundamental  to  conceptual  model  development,  which  itself  is  a 
crucial  activity  in  large  and  complex  projects. 

■  Is  not  a  new  idea,  though  it  is  (perhaps)  under-utilized  in  the  DoD 
community. 

■  Enables  discovery  of  relationships  between  entities  within  the 
domain  and  analysis  of  technical  problems. 

■  Results  in  a  robust,  relatively  invariant  model  applicable  across 
related  domains. 

■  Facilitates  linkage  of  diverse  projects  and  processes  into  a  unified 
portfolio. 

■  Increases  efficiency  of  acquisition  processes  through  flexibility  and 
reusability. 

■  Provides  a  common  foundation  for  M&S,  architecture,  analysis,  and 
project  management  tasks  . . . 

Convergence 

API 
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Sources 

ROBINSON,  S.  2004.  Simulation:  The  Practice  of  Model  Development 
and  Use.  John  Wiley  &  Sons,  Ltd,  West  Sussex,  England. 


ROSENBERG,  D.,  AND  SCOTT,  K.  2001.  Driving  Design:  The  Problem 
Domain.  Dr.  Dobb’s  Journal. 


ROSENBERG,  D.,  AND  STEPHENS,  M.  2007.  Use  Case  Driven  Object 
Modeling  with  UML.  Apress,  Berkeley,  CA. 
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Questions? 


Contact  Info: 

Nathaniel  Homer  Nathaniel.Horner@ihuapl.edu 

(240)  228-2908 

Steve  Topper  Steve.Topper@ihuapl.edu 

(240)  228-2701 


-  Carl  Sagan 
(1934-1996) 
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27  September  2007 
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Backups 
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Use  Cases 

What  it  is:  Descriptions  of  interactions  between  the  system  and  its  users. 

How  to  do  it: 

■  Identify 

■  The  actors  -  users  of  the  system,  including  other  systems. 

"  The  tasks  facilitated  by  the  system. 

■  The  actors’  participation  in  the  tasks,  including  alternate  courses  of 
events. 

■  Use  vocabulary  previously  defined  in  domain  model. 

■  Go  back  and  alter  the  domain  model  as  errors  are  uncovered  through  use 
case  exploration. 

What  it  is  for: 

■  Answers  the  question,  “What  are  the  user  experiences  with  the  system?” 

■  Helps  define  scope  and  provides  general  basis  for  more  formal  modeling. 

■  Provides  foundation  for  the  dynamic  behavioral  model. 
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Use  Case  (Example) 
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"  Use  cases  are  listed 
in  a  diagram 
showing  the 
participating  actors. 


■  Each  use  case  is 
expanded  into  a 
document 
describing  the  flow 
of  events  involved, 
including: 

■  Actors  involved 

■  Preconditions 

■  Event  sequences 

■  Exceptions 

■  Participants 

■  Alternatives 

■  Unresolved 
issues 
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Class  Model 

What  it  is:  A  more  detailed  static  representation  of  the  domain. 


How  to  do  it: 

■  Extend  the  domain  model. 

■  Allocate  behaviors  to  domain  model  entities  based  on  use  case 
descriptions. 

■  Add  attributes  and  operations  to  domain  model  entities. 

■  Add  classes  to  the  solution  space  as  necessary. 

■  Work  iteratively,  going  back  and  forth  between  static  model  and 
behavioral  model  (e.g.,  activity,  sequence  diagrams). 


What  it  is  for:  Begins  to  translate  general  descriptions  into  more 
formal  system  design. 
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UNCLASSIFIED 


UNCLASSIFIED 


Class  Model  (Example) 


UNCLASSIFIED 

Activity,  State,  other  Behavioral  Diagrams 

What  it  is:  A  more  detailed  dynamic  representation  of  the  system. 

How  to  do  it: 

■  Create  activity  diagrams: 

■  Break  up  use  cases  into  component  transactions  or  activities. 

■  Sequence  the  activities. 

■  Assign  responsibility  for  each  activity  to  a  domain  entity  via  swimlanes. 

■  Create  state  diagrams: 

■  Define  atomic  states  for  each  domain  entity. 

■  Sequence  the  states. 

■  Define  conditions  and  constraints  governing  state  transitions. 

■  Use  the  use  cases  as  a  primary  input. 

■  Work  iteratively,  going  back  and  forth  between  the  behavioral  model  and  the 
static  model  (domain  and  class  model),  ensuring  compatibility. 

What  it  is  for:  Begins  to  formalize  use  cases  into  more  detailed  system 
behaviors  and  activities. 

dPL 
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Behavioral  Model  (Example) 


UNCLASSIFIED 


O  State  Diagram 
C  Activity  Diagram 
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The  Advance  Man  Portable  Air  Defense 
System  (A-MANPADS)  allows  the 
Marines  of  Low  Altitude  Air  Defense 
(LAAD)  battalions  to  successfully  meet 
their  primary  mission. 

-  Marine  Corps  LAAD  units  deploy  in  one 
of  two  primary  missions;  convoy 
support  or  local  area  defense.  In  both 
roles,  LAAD  units  provide  primary  air 
defense. 

The  A-MANPADS  provides  a  means  to 
safely  and  expeditiously  transport  4 
Stinger  missiles  in  WRCs  and  ancillary 
equipment. 

The  installation  of  the  weapons  station 
allows  the  Marines  the  option  of 
mounting  a  crew  served  weapon  such 
as  the  7.62  machine  gun,  M240B,  or  the 
.50-caliber  machine  gun,  M2  Heavy 
Barrel  (HB).  The  crew  served  weapon 
could  be  utilized  for  self-protection 
against  both  air  and  ground  threats 
within  the  inner  launch  boundary  of  the 
missile. 


A-MANPADS  with  240B  Machine  Gun 
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In  the  case  of  the  A-MANPADS, 
the  crew  served  weapon  is 
flexible  therefore  the  pintle 
needs  to  be  flexible. 

-  The  Mk  93  Universal  Pintle 
provides  the  ability  to  switch 
between  all  crew  served 
weapons  in  the  Marine  Corps’ 
arsenal  with  a  minimum  of 
effort. 

-  The  Mk93  includes  an 
adjustable  safety  stop  for 
restricting  the  depression  angle. 
This  allows  the  pintle  to  not 
only  adjust  depending  upon  the 
weapon  system,  but  also  the 
vehicle  load  out. 


Safety  Stop 


Mk  93  Pintle  Installed  on  an  A-MANPADS 
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•  The  Mk  93  pintle  utilized  with 
the  HMMWV  weapons  station 
and  a  crew  served  weapon 
allows  for  a  maximum 
declination  angle  of  27°.  In 
the  standard  configuration 
with  the  Ml  025/MI  043  slant- 
back  HMMWV,  this  angle 
does  not  present  an  issue. 
The  trajectory  of  the  round 
would  pass  through  the 
HMMWV  outer  shell  in  an 
area  where  no  gear  is  stowed. 
However,  the  addition  of  the 
WRCs  adds  height  to  the  rear 
dimension.  If  allowed  to  fire 
at  maximum  depression  the 
round  would  impact  the  WRC 
as  demonstrated  by  the 
figure. 


Trajectory  of  Crew  Served  Weapon  Round 
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•  The  methodology  used  to  classify  and 
rank  mishap  risks  is  based  upon 
criteria  and  guidelines  specified  in  MIL- 
STD-882 

-  A  combat  loaded  A-MANPADS  is  valued  at 
less  than  $300k.  With  this  in  mind  the 
dollar  values  were  removed  and  system 
damage  was  evaluated  with  the  MIL-STD- 
882C  criteria. 

•  A  group  of  independent  system  safety 
engineers  determined  that  an 
impingement  incident  was  both 
catastrophic  and  likely  to  occur  several 
times  during  the  life  of  the  A- 
MANPADS. 

-  The  Hazard  was  assessed  a  Risk  Level  of 
High,  1C.  Thus  requiring  the  Assistant 
Secretary  of  the  Navy  to  accept  the  risk. 

•  The  Program  Manager  requested  an  in- 
depth  review  of  the  Hazard. 


HAZARD  RISK  INDEX 

RISK 

LEVEL 

ACCEPTANCE  AUTHORITY 

I A/B/C,  II A/B,  IIIA 

High 

Component  Acquisition  Executive 
(Assistant  Secretary  of  the  Navy  for 

Research,  Development  and  Acquisition) 

I  D,  II  C,  III  B 

Serious 

Program  Executive  Officer  (Commanding 
General,  Marine  Corps  Systems  Command) 

I E,  II  D/E,  III  C/D/E, 

IV  A/B 

Medium 

Program  Manager  (Program  Manager,  Air 
Defense  Weapon  Systems) 

IV  C/D/E 

Low 

Program  Manager  (Program  Manager,  Air 
Defense  Weapon  Systems) 

Risk  Acceptance  Levels  as  stated  in  MIL-STD-882C 
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•  An  accurate  assessment  of  the 
severity  of  a  round  striking  a 
Stinger  missile  can  be  garnered 
from  a  simple  evaluation  of  the  end 
results. 

-  The  Stinger  Missile  costs  less  than 
$100k 

-  The  missile  is  a  mission  critical 
component. 

•  If  the  missile  is  rendered 
inoperable,  the  A-MANPADS 
becomes  non-mission  capable, 
temporarily  resulting  in  a  de 
facto  combat  loss. 

•  The  Hazard  is  assessed  a  Severity 

of  Category  I,  Catastrophic. 


SEVERITY 

CATEGORY 

RESULT  CRITERIA 

Catastrophic 

1 

Could  result  in  death,  permanent  total  disability, 
system  loss,  or  irreversible  severe  environmental 
damage  that  violates  law  or  regulation. 

Critical 

II 

Could  result  in  permanent  partial  disability, 
injuries  or  occupational  illness  that  may  result  in 
hospitalization  of  at  least  three  personnel,  major 
system  damage,  or  reversible  environmental 
damage  causing  a  violation  of  law  or  regulation. 

Marginal 

III 

Could  result  in  injury  or  occupational  illness 
resulting  in  one  or  more  lost  workdays,  minor 
system  damage,  or  mitigatible  environmental 
damage  without  violation  of  law  or  regulation 
where  restoration  activities  can  be  accomplished. 

Negligible 

IV 

Could  result  in  injury  or  illness  not  resulting  in  a 
lost  workday,  less  than  minor  system  damage,  or 
minimal  environmental  damage  not  violating  law 
or  regulation. 

Mishap  Severity  Categories  as  stated  in  MIL-STD-882C 
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•  The  original  Safety 
Analysis  assessed  a 
Probability  level  of  C, 
Occasional,  based  on 
the  following  criteria: 

-  Properly  setting  the 
adjustable  depression 
stop  is  a  training  issue. 

-  Training  issues  are  a 
result  of  human  error. 

-  Human  error  has  a 
probability  of  1  x  10'3 


DESCRIPTIVE 

WORD 

LEVEL 

INDIVIDUAL  ITEM 

FLEET  OR 

INVENTORY 

Frequent 
(X>  101) 

A 

Likely  to  occur  frequently 

Continuously  experienced 

Probable 
(101  >X>  10-2) 

B 

Will  occur  several  times  in 
life  of  an  item 

Will  occur  frequently 

Occasional 
(10 2>X>  10-3) 

c 

Likely  to  occur  sometime  in 
life  of  an  item 

Will  occur  several  times 
across  fleet 

Remote 

(10'3>X>  10'6) 

D 

Unlikely,  but  possible  to 
occur  in  the  life  of  an  item 

Unlikely,  but  can 
reasonably  be  expected  to 
occur 

Improbable 
(10-6  >  X) 

E 

So  unlikely,  it  can  be 
assumed  occurrence  may 
not  be  experienced 

Unlikely  to  occur,  but 
possible 

Mishap  Probability  Levels  as  stated  in  MIL-STD-882C 
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•  Weapon  System  Explosives  Safety  Review 
Board  (WSESRB)  has  stated 


“programs  need  to  be  utilizing 
one  of  the  various  methods  (of 
human  error  prediction)  and  not 
use  a  blanket  number  (1  x  10~3)” 

Human  Error  Quantification ,  WSESRB  Executive  Session ,  November  2005 
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•  The  original  assessment  had  only  considered  the 
final  action  that  would  lead  to  the  mishap. 

•  Assessing  a  probability  of  failure  for  a  situation 
starts  by  determining  the  series  of  actions  that  the 
operator  undertakes  for  the  particular  situation. 
The  methodology  to  determine  the  actions  is 
known  as  a  fault  tree  analysis  (FTA). 

•  An  FTA  begins  with  the  selection  of  an  undesirable 
outcome,  the  root.  Then,  each  situation  that  could 
cause  that  outcome  is  added  to  the  tree.  Further 
branches  are  added  by  assessing  possible  causes 
for  each  successive  layer  of  contributing  factors. 
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•  The  Human  Factors  Analysis  and 
Classification  System  (HFACS)  was 
selected  for  the  A-MANPADS  due  to  the 
inclusion  of  environmental,  psychological, 
emotional,  and  physical  influences  on  the 
operator,  in  addition  to  the  active  faults  of 
the  operator. 

•  HFACS  was  originally  developed  by  the 
Federal  Aviation  Administration  (FAA)  and 
has  been  adopted  by  the  US  Navy  for 
investigating  the  underlying  reasons  for 
human  error  in  aviation  accidents. 

•  HFACS  was  developed  based  on  the  “Swiss 
Cheese”  model  of  human  error  described 
by  James  Reason  (Reason,  1990).  Most 
investigations  only  focus  on  the  operator’s 
final  error(s)  that  lead  to  the  mishap. 
However,  the  “Swiss  Cheese”  model  states 
that  it  is  the  alignment  of  many  factors  at 
many  levels  of  the  organization  that  align 
perfectly  to  allow  or  lead  to  the  final  error, 
much  like  the  holes  of  many  layers  of  Swiss 
cheese  aligning  to  allow  light  through. 


Reason  ’’Swiss  Cheese’’  Model 
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Not  only  were  the 
actions  of  the  Marine 
firing  the  weapon 
evaluated,  but  also 
the  preexisting 
environmental 
conditions  and  the 
organizational 
doctrine  required  to 
initiate  the  chain  of 
events. 


Marine  Fires  Weapon  into  WRC 
Act 
Error 

Skill  Based  Error 


Marine  Improperly  Checks  Mechanical  Stop 
Act 

Violation 


Marine  Improperly  Sets  Mechanical  Stop 
Act 

Violation 


A-MANPADS  Attacked  While  on  Mission 
Preconditions 
Physical  Environment 
Unit  Mission 


A-MANPADS  Sent  on  Mission  with  Live  Missiles 
Organizational 
Organizational  Climate 
Unit  Mission 


A-MANPADS  Sent  to  Combat  Zone 
Organizational 
Organizational  Climate 
Unit  Mission 
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•  Human  Error  Assessment  and  Reduction  Technique  (HEART)  Method 

-  The  HEART  Method  provides  two  tables  to  find  the  human  error  rate.  A  factor  from  the  first  table 
is  multiplied  by  chosen  factors  from  the  second  table. 

-  Based  on  expert  opinion  -  cannot  be  validated 

-  There  is  the  difficulty  in  dealing  with  the  many  variables  which  contribute  to  the  probability  of 
error  occurrence  at  any  point  in  time. 

•  SPAR-H  Method 

-  Provides  a  simple  worksheet  with  multipliers  for  stress,  complexity,  experience,  etc. 

-  Computationally  intensive 

•  Operator  HEP  Estimate 

-  The  Reactor  Safety  Study  lists  Operator  Human  Error  Probability  (HEP)  Estimates  for  each 
scenario  description 

-  Has  a  limited  number  of  scenarios.  Expert  judgment  must  be  used  in  selecting  a  scenario  that 
can  be  used  as  a  substitute. 

•  Human  Reliability  Table 

-  Lists  Operator  HEP  Estimates  for  each  general  scenario  description. 

-  Generalized  scenarios  limit  fidelity.  Expert  judgment  must  be  used  in  selecting  a  scenario  that 
can  be  used  as  a  substitute. 

•  WSESRB  Guidebook  Worksheets 

-  Supply  complex  tables  of  factors  that  take  into  account  fatigue,  stress,  training,  complexity,  etc. 
These  factors  are  used  in  a  series  of  binomial  equations  which  derive  a  final  error  rate. 

-  Computationally  intensive 
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•  The  Standardized  Plant  Analysis  Risk  Human  Reliability  Analysis  (SPAR-H) 
developed  by  the  US  Nuclear  Regulatory  Commission  (NRC)  takes  into  account 
performance  shaping  factors  (PSFs).  SPAR-H  makes  allowance  for  the 
following  factors: 

-  Available  Time 

-  Stress  and  Stressors 

-  Experience  and  Training 

-  Complexity 

-  Ergonomics 

-  Procedures 

-  Fitness  for  Duty 

-  Work  Processes 

•  Not  only  does  SPAR-H  account  for  a  greater  number  of  influences,  but  it  also 
takes  into  account  positive  benefits  derived  from  some  PSFs. 

•  SPAR-H  makes  a  distinction  between  diagnosis  (i.e.,  the  processing  of 
information)  and  action  (i.e.,  the  response). 

•  It  assigns  a  base  value  to  the  HEP  for  basic  processes.  A  multiplier  for  each  of 
the  eight  PSFs  is  then  factored  into  determining  the  overall  HEP. 

•  SPAR-H  allows  for  the  occasion  where  the  diagnosis,  and  the  action  are  so 
interrelated  that  they  can  not  be  separated.  Likewise,  SPAR-H  includes  a 
correction  factor  for  cases  where  the  influence  of  PSFs  is  so  great  that  an 
inaccurate  HEP  is  produced. 
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Available  Time:  Available  time  refers  to  the  time 
the  operator  has  to  make  a  diagnosis  and  act  upon 
the  diagnosis.  When  time  is  short  an  operator 
tends  to  analyze  fewer  possible  alternatives. 

Stress/Stressors:  Stress  is  broadly  defined  as 
motivating  forces  that  have  both  positive  and 
negative  effects  on  human  performance.  Small 
amounts  of  stress  can  lead  to  increased  work 
performance,  however,  as  the  level  of  stress 
increases  the  ability  to  successfully  complete 
tasks  decreases. 

-  The  previous  work  that  SPAR-H  derived  from 

allowed  a  multiplier  of  25  when  the  operator  believed 
himself  to  be  in  a  life-threatening  situation.  When  in 
combat  the  operator  knows  that  he  is  in  a  life- 
threatening  situation.  Therefore  a  multiplier  of  25 
will  be  utilized  for  combat  situations. 

Complexity:  Complexity  incorporates  both  the 
difficulty  and  the  ambiguity  of  a  task.  If  the  task  is 
mentally  or  physically  difficult  to  perform  the 
likelihood  of  failure  increases  noticeably. 

Experience/  Training:  Formal  schooling,  on  the  job 
training,  years  of  experience  with  the  system,  and 
previous  exposure  to  similar  events  are  all  factors 
taken  into  consideration  when  determining  the 
value  of  this  PSF. 


Category 

SPAR-H  Value 

Combat 

Adjustment 

Inadequate 

Failure 

Time  Available 
=  Time 
Required 

10 

Available  Time 

Nominal  Time 

1 

N/A 

Time  Available 
>  5x  Time 
Required 

0.1 

Time  Available 
>  50x  Time 
Required 

0.01 

Extreme 

5 

Stress/  Stressors 

High 

2 

25 

Nominal 

1 

Highly  Complex 

5 

Complexity 

Moderately 

Complex 

2 

N/A 

Nominal 

1 

Low 

3 

Experience/  Training 

Nominal 

1 

N/A 

High 

0.5 
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Procedures:  This  PSF  accounts  for  the  existence 
and  usage  of  formalized  procedures. 

Ergonomics:  Ergonomics  considers  the  ease  of 
interaction  between  the  human  and  the  machine. 

Such  factors  include,  availability  of 
instrumentation,  positioning  of  instrumentation, 
ease  of  understanding  the  information  presented, 
and  the  layout  of  the  controls. 

Fitness  for  Duty:  This  PSF  considers  the  physical 
and  mental  capacity  of  the  operator  to  properly 
perform  the  task.  Considerations  include  drug 
usage,  illness,  fatigue,  distractions,  and  personal 
problems. 

-  While  combatants  are  generally  physically  fit,  the 
conditions  surrounding  combat  not  only  equalize 
this  advantage  but  often  degrade  the  fitness  of  the 
operator  beyond  that  of  a  fever  or  some  cough 
syrup.  To  account  for  this  a  multiplier  of  10  is 
utilized  for  combat  situations. 

Work  Process:  Work  Process  captures  the 
company  culture  and  “way  of  doing  business”.  It 
considers  how  the  work  is  planned  and 
communicated,  how  management  supports  or 
enforces  policies,  and  how  the  company  as  a 
whole  values  safety,  quality,  and  the  individual 
worker. 
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Category 

SPAR-H  Value 

Combat  Adjustment 

Procedures 

Not  Available 

50 

N/A 

Incomplete 

20 

Available  but 
Poor 

5 

Nominal 

1 

Ergonomics 

Missing/ 

Misleading 

50 

N/A 

Poor 

10 

Nominal 

1 

Good 

0.5 

Fitness  for  Duty 

Unfit 

Failure 

10 

Degraded 

Fitness 

5 

Nominal 

1 

Work  Process 

Poor 

2 

N/A 

Nominal 

1 

Good 

0.8 -0.5 
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HEP, 


No  min  al 


=  HEPBase  •  PSF„ . .  PSF, . .  PSF,, . .  PSF,,..,..  •  PSF,, . .  PSF . .  PSF,,.  •  PSF, 


Time 


Stress 


Comp 


Train 


Pr  oc 


HMI 


Fit 


Work 


•  The  multipliers  are  utilized  by  multiplying  the  base  HEP  for  action  or 
diagnosis  by  the  8  PSF  multipliers. 

•  The  Base  Multipliers  are: 

-  0.01  for  diagnosis 

•  The  user  is  required  to  decide  what  the  correct  action  should  be  based  on  external 
stimuli. 

-  0.001  for  action 

•  The  user  implements  the  action  as  stated  in  a  procedure  or  that  they  have  chosen 
based  on  their  diagnosis. 

•  If  the  PSFs  are  significantly  negative,  the  HEP  can  become  inordinately 
large.  To  help  adjust  the  HEP  in  the  event  of  overwhelming  negative 
influences  a  simple  mathematical  formula  is  provided  below: 


HFP 

Adjusted 


MFP  •  PC/7 

* No  min  al  ^  Composite 


PfPP  • 

11J^1  Nominal 


{PSFCo  site  lj 


+  1 
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*  Conservative  assumptions  made. 

-  All  armored  A-MANPADS  and  only 
armored  A-MANPADS 

-  Used  a  quarter  of  their  life  cycle  in 
combat 

-  Loaded  with  live  missiles  half  of 
the  time. 

-  Under  attack  every  time  they  went 
to  combat 


R 


AMANPADS 


Armored 


Combat 


AMANP ADSTotal 


%  P  #  P  #  P 

Life  Missiles  Attack 


0.0266 


40 

188 


0.25*  0.5*1 


2.66%  chance  of  an  A-MANPADS  transporting 
missiles  while  being  attacked 
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•  Just  because  the  Marine  returns  fire  does 
not  guarantee  the  rounds  are  traveling 
towards  the  missiles. 

•  In  lieu  of  data  representing  the  number  of 
attacks  to  the  rear  of  vehicles,  the 
percentage  of  the  area  on  the  vehicle 
considered  to  be  the  danger  zone  will  be 
calculated. 

-  The  assumption  is  made  that  the  operator 
never  fires  the  machine  gun  elevated. 


DZ  az  •  DZel 
360°  22° 


0.0621 


41°  12° 

360°  *22° 


6.21%  chance  of  being 
attacked  from  the  rear 
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•  When  the  armorer  receives  a 
new  pintle,  a  new  mission  role 
with  a  load  out  that  requires  a 
depression  angle  change,  or  a 
misaligned  pintle  is  returned,  the 
armorer  sets  the  depression 
angle. 

•  a  0.005%  chance  that  the 

armorer  will  fail  to  complete  the 
adjustment  is  reasonable. 

-  It  is  a  required  step  of  a 
procedure,  ample  time  is 
supplied  to  complete  the 
process,  a  follow  on  procedure 
performed  by  an  independent 
person  checks  for  the 
completion  of  this  task,  and  the 
steps  are  well  documented  and 
simple. 
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Category 

Level 

Value 

Reason 

Base  HEP 

Action 

0.001 

Action  only 

Available  Time 

5x  Req 

0.1 

Armorer  completes  the  task  offline 
with  more  than  ample  time. 

Stress/  Stressors 

Nominal 

1 

With  ample  time  to  complete  and 
no  dependency  on  outcome, 
armorer  is  not  stressed. 

Complexity 

Nominal 

1 

Steps  are  straight  forward  and 
easy  to  follow 

Experience/  Training 

Nominal 

1 

The  job  is  simple  but  the  armorer 
only  does  it. 

Procedures 

Nominal 

1 

The  procedure  is  well  documented 
and  clearly  written. 

Ergonomics 

Nominal 

1 

Ergonomics  neither  impede  nor 
help 

Fitness  for  Duty 

Nominal 

1 

The  armorer  is  more  than  fit 
enough. 

Work  Process 

Good 

0.5 

The  expectations  are  well  defined 
and  communicated  clearly. 

Nominal  HEP 

0.00005 
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As  the  Marine  is  installing  the 
pintle  and  the  machine  gun,  the 
procedures  instruct  the  Marine  to 
check  the  depression  angle  using 
available  components  and  tools. 

-  The  Marine  is  instructed  to  alert 
the  armorer  if  the  pintle  is 
misaligned. 

-  Before  leaving  on  the  mission,  the 
senior  Marine  in  the  vehicle 
ensures  that  preoperational 
checks  were  preformed. 

A  0.05%  chance  that  the  operator 
will  fail  at  the  check  is  reasonable. 

-  It  is  a  required  step  of  a  procedure 
completed  often,  a  person  in  a 
supervisory  role  checks  for 
completion,  and  the  steps  are  well 
documented  and  simple. 


Category 

Level 

Value 

Reason 

Base  HEP 

Action 

0.001 

Action  only 

Available  Time 

Nominal 

1 

Part  of  the  installation  of  the 

weapon  and  sufficient  time  is 
provided 

Stress/  Stressors 

High 

2 

Operator  is  preparing  for  combat, 
anticipation  and  fear  begin  to 
increase  stress 

Complexity 

Nominal 

1 

Steps  are  straight  forward  and  easy 
to  follow 

Experience/  Training 

High 

0.5 

The  same  procedure  is  followed 
every  time  the  weapon  is 
installed 

Procedures 

Nominal 

1 

The  procedure  is  well  documented 
and  clearly  written. 

Ergonomics 

Nominal 

1 

Ergonomics  neither  impede  nor 
help 

Fitness  for  Duty 

Nominal 

1 

The  operator  may  be  uncomfortable 
but  their  fitness  is  not 
degraded. 

Work  Process 

Good 

0.5 

The  expectations  are  well  defined. 
Additionally  the  supervisor 
ensures  that  the  process  is 
completed. 

Nominal  HEP 

0.0005 
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CRANE 


When  the  Marine  identifies 
a  threat  and  begins  firing, 
there  is  a  probability  that 
he  will  continue  to  fire 
even  if  the  rounds  are 
going  to  impact  the  WRC. 

An  adjusted  value  of 
47.39%  is  a  reasonable 
percentage  to  expect. 

-  When  in  combat  and 
under  attack,  operators 
are  likely  to  experience 
tunnel  vision  and  fixate 
on  the  threat  until  it  is 
eliminated. 

-  The  adjustment 
equation  was  utilized  to 
correct  for  the 
overwhelming 
multipliers. 


Category 

Level 

Value 

Reason 

Base  HEP 

Action 

0.001 

Action  only 

Available  Time 

Avail  =  Req 

10 

In  combat  Oper.  Always  has  just 
enough  time 

Stress/  Stressors 

Combat 

25 

Life  threatening  situation 

Complexity 

Nominal 

1 

Firing  the  weapon  is  relatively  easy 

Experience/ 

Training 

Above  Avg 

0.6 

Even  the  newest  member  of  the 
squad  trains  on  the  system 
rigorously.  However,  rear 
attacks  and  shooting  around 
the  WRC  are  not  well  rehearsed. 

Procedures 

Nominal 

1 

Procedures  are  well  established  and 
followed  explicitly. 

Ergonomics 

Nominal 

1 

Ergonomics  neither  impede  or  help 

Fitness  for  Duty 

Combat 

10 

Even  the  most  physically  fit 
personnel  suffers  from 
degradation  of  fitness  in 
combat 

Work  Process 

Above  Avg 

0.6 

While  fog  of  war  impedes  the  process; 
expectations  are  clear,  concise, 
well  communicated,  and 
strictly  enforced. 

Nominal  HEP 

0.9 

Adjusted  HEP 

0.47393365 

Due  to  the  large  number  of  negative 
multipliers  the  adjustment  was 
used. 
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*  WRC  Shot  When  No  Depression  Stop  is  Present: 

-  The  A-MANPADS  is  in  combat 

-  The  weapon  enters  the  “danger  zone” 

-  The  Marine  fires  the  weapon  while  in  or  around  the  “danger  zone”. 


P 


NoStop 


=  P 


Combat 


•  HEP ; 


DZ 


Shoot 


7.8301  xlO"4  =  2.6596 xlO"2  •6.2121  xlO"2  •  4.7393  xlO"1 


With  no  stop  present  and  conservative  representations  of  the  likelihood 
of  the  A-MANPADS  being  in  combat  with  a  live  missile  and  attacked  from 

behind,  the  probability  of  shooting  the  WRC  is  7.8301x1  O'4  or  7.83 
chances  in  one  thousand. 
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CRANE 


*  The  Depression  Stop  is  Misaligned  by  the  Marine: 

-  The  A-MANPADS  is  in  combat 

-  The  weapon  enters  the  “danger  zone” 

-  The  Marine  fires  the  weapon  while  in  or  around  the  “danger  zone”. 

-  The  operator  misaligns  the  pintle 


P  —  P  •  PfPP 

Check  NoStop  Check 

3.9151  xlO-7  =  7.8301  x  10"4  •  5  x  10-4 


With  the  addition  of  a  depression  stop  the  probability  of  shooting  the 
WRC  is  3.91 51x1  O'7  or  approximately  one  in  250,000. 


24 


Harnessing  the  Power  of  Technology  for  the  Warfighter 


CRANE 


*  The  Depression  Stop  is  Misaligned  by  the  Armorer: 

-  The  A-MANPADS  is  in  combat 

-  The  weapon  enters  the  “danger  zone” 

-  The  Marine  fires  the  weapon  while  in  or  around  the  “danger  zone”. 

-  The  armorer  misaligns  the  pintle  or  fails  to  align  the  pintle  at  all 

-  The  operator  does  not  find  the  misalignment. 


P 


Misalign 


=  R 


Check 


HEP 


Misalign 


1.9575 xlO-11  =  3.9151  x  10-7  »5xl0_5 


By  making  the  armorer  responsible  for  the  adjustment  of  the  safety  stop, 

the  probability  of  shooting  the  WRC  becomes  1.9575x1  O'11  or 
approximately  one  in  50  Billion. 
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Scenario 

Severity 

Probability 

Acceptability 

Authority 

No  Depression  Stop 

I 

D 

ID 

Program  Executive  Officer 

Adjustable  Depression  Stop 

I 

E 

IE 

Program  Manager 

Armorer  Adjusts  Pintle 

I 

E 

IE 

Program  Manager 

•  The  risk  associated  with  the  A-MANPADS  operating  with  the  adjustable 
stop  provided  with  the  Mk  93  pintle  is  of  a  level  acceptable  by  the  Program 
Manager. 

-  Based  upon 

•  The  condition  that  all  controls  and  procedures  are  complied  with 

•  The  A-MANPADS  will  be  operated  within  stated  parameters 
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CRANE 

•  After  the  study  was  conducted,  the  Program  Manager  was 
able  to  accept  the  risk  associated  with  the  hazard,  the 
Program  received  a  full  rate  production  decision,  and  all 
systems  were  fielded  on  schedule. 

•  The  use  of  a  fault  treat  analysis  such  as  HFACS  for  Safety 
Assessment  Probability  Levels  is  crucial  to  capturing  a  true 
picture  of  all  the  factors  leading  to  a  hazard. 

•  While  the  use  of  SPAR-H  requires  computational  effort,  I  have 
demonstrated  that  the  math  is  uncomplicated  and  relatively 
concise. 

•  SPAR-H  includes  the  flexibility  to  be  utilized  for  any  Program. 
It  does  not  depend  upon  predetermined  scenarios,  but  rather 
considers  8  performance  shaping  factors  that  are  crucial  to 
success  in  any  action  or  diagnosis. 

•  With  the  comparative  ease  of  applying  SPAR-H,  there  is  no 
need  for  a  program  to  arbitrarily  apply  a  blanket  number  (1  x 
10' )  to  their  Safety  Assessment  Probability  Levels. 
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(U)  UNCLASSIFIED 


Questions? 


(E  CENTERS 


Modeling  and  Simulation 

Resource  Reuse 
Business  Model 


Dennis  P.  Shea 


CNA 

ANALYSIS  &  SOLUTIONS 


Outline 


•  Problem  statement 

Inefficient  use  of  M&S  resources 

Barriers  to  reuse 

Multiple  perspectives  on  reuse 

•  Study  approach 

•  Review  federal  laws,  DoD  regulations  and  policies  on  intra¬ 
government  business  transactions 

•  M&S  may  contain  intellectual  property 

•  Proprietary  M&S  and  reuse 

•  Lessons  learned  from  successful  M&S  reuse 

•  Framework  for  a  business  model 
Business  model  actions  to  spur  reuse 


CNA 


The  Problem: 

Inefficient  Use  of  M&S  Resources 


Few  M&S  resources  are  reused  -  either  during  a  single 
program’s  lifecycle  or  across  acquisition  programs. 


Tools 

-  Models 

-  Simulations 

-  Federations 

-  Utilities  (post¬ 
processors) 


Data 


-  Input  datasets 

-  Scenarios 
- CONOPs 

-  Threat  data 
-Algorithms 

-  Environmental 
info 


Environment 

-Architectures  -  Network  resources 

-  Interfaces  -  SME  expertise 

-  Protocols 

-  VV&A  templates 


Absence  of  incentives  for  Gov’t  M&S 
managers  and  industry  developers 


CNA 
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Reuse  doesn't  mean  necking  down  to  a  single 
model  or  database 


Spectrum  (Visual,  IR) 


Latency  (Real-time,  NRT) 


Accuracy  (+/-  deg,  pixels) 


Format  (DTED,  Open  Flight) 


Data  Source  (commercial) 


Classification  (U,  S,  TS) 


Dimensionality  (2D,  3D) 


Area  (NM,  sq.  blocks) 


Display 

(Height/width  scene) 


Resolution  (pixels,  degrees) 


CNA 


Barriers  to  M&S  Resource  Reuse 


Users  lack  awareness  of 
reusable  resources 

Insufficient  details  about 
reusable  resources 

Hard  to  assess  the  true 
capabilities  and  limitations 
of  existing  resources 

Resources  not  in  a  form 
suitable  for  reuse 

Users  lack  trust  in 
resources  developed  by 
others/  NIH 

Model  is  available  but  not 
the  data 

M&S  components  don’t 
work  well  together 


Repositories  are  incomplete 
and  not  current 

*  Little  insight  into  how 
resources  have  been  used  in 
the  past,  including 
successfully  and  failures 

*  Difficult  to  access  the  actual 
resource 

*  Difficult  to  adapt  existing 
resources  to  new  problems 

*  No  mechanism  to 
compensate  developer  for 
resource  investment  and 
guidance  on  use 

*  No  mechanism  to  protect 
developer  from  mischievous 
uses 


CNA 
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Multiple  perspectives  on  M&S  reuse 


Program  Manager 


Industry 

CNA 


DoD  (writ  large) 
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Objective 


•  Develop  an  economic  business  model  that  will  make 
the  reuse  of  M&S  resources  an  attractive  option  for 
both  consumers  and  providers  of  resources 

-  Puts  the  best  M&S  resources  in  the  hands  of 
users 

Fosters  collaboration  and  sharing 

Leads  to  cost  efficiency  and  minimal  duplication 
of  effort 

-  Protects  IP  rights  of  industry 

-  Ensures  profitability  of  M&S  industry 


CNA 
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M&S  Reuse  Puzzle 


CNA 


Approach 


*  Reviewed  existing  policy  documents,  DoD 
instructions,  guidance,  interagency  agreements, 
FAR,  DFARS,  prior  reports,  ... 

*  Prepared  case  studies 

-  SIMDIS,  Linux,  EADSIM,  ICT,  NIH/OTT,  ... 

*  Used  a  variety  of  survey  instruments,  interviews,  e- 
mail  dialogue  with  industry  and  government 

Where  is  reuse  occurring  today? 

What  “business  factors”  help  to  motivate  reuse? 

What  are  the  challenges  to  reuse  and  how  might 
these  be  overcome? 


CNA 
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Who  we  have  spoken  to 


—  • 


Northrop  Grumman 
Aegis  Technology 
MAK  Technologies 
PM  FCS  AD  M&S 
NGA 

NAVAIR  Portable  Source  Initiative 

OSD-JDS 

BreakAway,  LTD 

MSIC,  DIA  TMAP 

USJFCOM  J9 

USAF  Common  Data  Set 

M&S  EA  (Ocean,  Air&Space,  Terrain) 

IWS  General  Council  (SEAOO) 

Pitch  Technologies 
MMA  M&S 


Boeing 

Soar  Technology 

Lockheed  Martin 

MOVES/NPS 

Metron 

NAVMSMO 

JSF  M&S 

IWS  M&S 

USN  IWS  SHARE 

SAF/XC 

OPNAV  N814 

NRL 

JASP 

MSIAC 


CNA 
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Key  issue  for  a  business  model 


Under  what  conditions  can  a  DoD 

program  manager  or  other  government 

official  invest  in  M&S  today 

-  to  satisfy  both  current  and  future 
requirements, 

-  including  perhaps  requirements  of  another  yet 
unknown  government  user, 

-  Including  additional  investment  to  make  the 
M&S  resource  reusable, 

-  and  be  compensated  in  a  future  intra¬ 
government  business  exchange? 


CNA 
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Federal  laws  and  DoD  regulations  affecting 
intra-government  business  transactions 


DoD 

FMR 

7000.14-R 
Vol.  11 A 


■QM 


DoD 

FMR 

7000.14-R 
Vol.  11 B 


■SI 


OMB 

Circ. 

A-130 


CNA 


DoD 

Instruc. 

4000.19 


Assessment 


Can’t  use  current  year  funds 
for  future  anticipated,  but 
unrealized  requirements 

Can’t  use  appropriations  for 
costs  to  be  reimbursed 
through  business  transactions 

Can’t  charge  for  costs  built 
into  budget 

Reimbursement  only  for 
marginal  costs 

May  charge  only  to  recover 
cost  of  dissemination 

May  transfer  asset  to  a 
working  capital  fund  and 
subsequently  charge  fully 
loaded  costs 
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Interagency  Acquisition  of  M&S 


Servicing  agency 

(1)  Existing  GOTS  or  COTS 
with  Gov’t  Purpose  Rights 


(2)  Same  as  (1)  +  Gov  personnel 
or  contract  support 

(3)  Same  as  (1)  + 
model  enhancements 

(4)  COTS  M&S  with  license 
requirements 

(5)  New  M&S  with  joint 
requirements 


Reguesting  agency 

(1)  No  compensation  allowed 

--  Congress  has  appropriated 
funds  to  servicing  agency 
-  No  increase  in  support 
supplier’s  costs 

(2)  Fund  incremental  cost 
of  labor 

(3)  Fund  model  enhancements 

(4)  Fund  incremental 
license  fees 

(5)  Jointly  fund  new  M&S 


M&S  resources  often  contain 

valuable  intellectual  property 

Intellectual  property  refers  to  creations  of  the  mind:  inventions,  literary 
and  artistic  works,  and  symbols,  names,  and  images  used  in  commerce. 

In  M&S  the  IP  is  often  encapsulated  in  the  source  code  and  data  sets 

DOD’s  access  to  M&S  IP  developed  under  contract  is  governed  by  both 
copyright  law,  patent  law,  and  the  procurement  regulations  contained  in 
the  DFARS 

These  laws  affect  the  Government's  ability  to  use,  reproduce,  modify, 
and  release  the  resource  to  one  or  more  potential  users 

Control  of  IP  is  determined,  in  part,  by  who  funded  development 
Government,  Industry,  or  Mixed 

But  formal  title  is  generally  retained  by  the  contractor-developer 
regardless  of  funding  source 

DoD  acquisitions  that  involve  a  mix  of  government  and  IRAD  funded 
technologies  pose  a  challenge  in  determining  control  “rights” 


14 


Proprietary  M&S  and  reuse 


COTS  provides  DoD  with  access  to  leading-edge  M&S  that 
otherwise  would  not  be  available 

•  COTS  supports  a  broader  market  than  DoD  and  thus 
capabilities  should  continue  to  improve  over  time 

But  a  challenge  to  maintain  legacy  systems 

•  COTS  enables  “agile”  M&S  investment  decisions  by 
eliminating  long-term  O&M 

•  Developer  may  earn  a  short-term  monopoly 

Until  the  next  wave  of  innovation 

•  Decouple  the  M&S  from  the  original  developer? 

Yes--  a  source  license  and/or  tech  data  rights  will  promote  3rd 
party  competition  and  encourage  DoD  to  develop  in-house 
talent  to  extend  the  M&S 

DoD  may  also  require  source  license  simply  to  “look  under 
the  hood” 

•  Enterprise  license  may  reduce  overall  DoD  costs  of  COTS 

•  Decision  on  negotiating  for  source  or  enterprise  license 
depends  on  reuse  potential  (and  willingness  of  developer) 


CNA 
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Lessons  learned  from 

successful  M&S  reuse 

-  • 


Keep  within  a  small, 
well-informed  community 

-  V&V  guides  use 

-  MOU  if  needed 


Discover 


Avoid 

misuse 


GPR  license  and  proper 
Sys  Eng  to  reduce 
costs 

Compensate 

MIPR  to  cover 
Additional  costs: 

-  Personnel 

-  Model  enhancements 

-  incremental  license  fee 


Keep  within  a  small, 
well-informed  community 


User / 
Provider 
relationship 


Assess 


Access 


Rely  on  standards 

-  V&V 

-  database  formats 
(OpenFlight,  Shape, 
GeoTIFF) 


Government 

Purpose 

Rights 


CNA 


Employ  Assume  knowledgeable  user 
-  User  funds  integration 
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M&S  Resource  Reuse  Business  Model 


M&S  Suppliers 
&  Support 
Infrastructure 


Partner  network 

-  Gov’t  agencies 

-  Labs 

-  Industry 

-  International 


Core  capabilities 

-  H/W  &  S/W 

-  System  information 

-  Org  &  Op  Knowledge 

-  Conceptual  models 


Value  activities 

-  Develop 

-  Test 

-  Validate 

-  Prototype 


A 


J 


f 


Value 

Proposition 

-  Savings  (time/$$) 

-  Authoritative 

-  Joint  context 

-  Interoperability 


V 


Compensation 

Distribution  channel 

-  Access  control 

-  Licensing 

-  IP  Intermediaries 

-  Royalties 

-MOUs 

-  Support  $$ 

-  Purchase  options 

Customer 


Target  Mkt 

-  PEOs,  PMs 

-  Dir  Training 

-  Hd  Analysis 

-  Service/Component 


Customer 

Relationships 

-  Discovery  tools 

-  Trust/  MOUs 


CNA 
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Business  model  actions  to  spur  M&S  reuse 


-  • 


Government 


Industry 


CNA 


Broadly 

Used 

Tools 


License 

Rights 


Open 
Business 
Models 


IP 


Open 

Standards 


Open 

Source 


< 


Centrally  fund 
Life  Cycle  Manage 

Track  development  history 
Assess  reuse  potential  early 
Manage  licenses 

M&S  intermediary 
Negotiate  licenses 
Registry  w/user  wiki 

Provide  volume  license 
NDAto  provide  visibility 

Specify  reuse  as  objective 
Fund  full  development  cost 
Specify  deliverables 

Transfer  resource  to  WCF 
Best  practices 

Interoperability 
Network  effects 

Field  initial  version,  then  OSS 
Rapid  test 
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Backup 
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Business  model  actions  that 
will  spur  M&S  reuse  (1  of  7) 

*  Improved  contracting  practices 

Specify  software,  tech  data,  documentation  as  a 
deliverable 

-  Price  contract  to  include  full  cost  of  making  M&S 

reusable  (licenses,  documentation,  V&V,  interfaces, ...) 

Include  expectations  for  software  reuse  in  solicitations 
(and  incentives  for  achieving  reuse) 

Implement  stronger  oversight  of  M&S  development 
process 

■  When  was  it  developed  and  who  paid  for  it? 

■  Is  contractor  entitled  to  restricted  or  limited  rights? 

■  Standard  contract  language  requiring  GPR  on  all 
datasets 

Require  registration  of  all  M&S  resources  (with 
metadata) 


CNA 
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-  • 


Business  model  actions  that 
will  spur  M&S  reuse  (2  of  7) 


*  Implement  improved  training  for  contract 
officers  and  program  managers 

Contract  Officers 

-  Goals  and  strategies  for  M&S  reuse 

-  Form  and  function  of  alternative  deliverables: 

Computer  programs,  source  code,  object  code,  algorithms,  flow 
charts,  computer  databases,  documentation,  etc. 

Program  managers  /  DoD  decision  makers 

-  Goals  and  strategies  for  M&S  reuse 

-  Software  licenses  and  tech  data  rights: 

Unlimited,  limited,  restricted,  government  purpose, 
commercial  license,  nonstandard  rights 

-  Negotiating  strategies 

•  Develop  a  “Best  Practices  Guide”  for 
contracting  M&S  resources 


CNA 
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Business  model  actions  that 
will  spur  M&S  reuse  (3  of  7) 

•  For  broadly  used  GOTS  M&S,  use  central 
funding  to  make  the  resource  reusable 
and  to  manage  Life  Cycle  Costs 

-  No  single  organization  can  be  responsible 

•  Similar  approach  for  common  databases 

-  Environment,  threat  models,  scenarios, 
current  and  future  forces  (Blue,  Red,  White) 

•  Negotiate  volume  or  enterprise  license 
for  proprietary  M&S 


CNA 
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Emerging  tenets  for  an 

M&S  business  model  (4  of  7) 

M&S  intermediary  to  create  a  secondary  market 


Patterned  after  IP  intermediary  (Innovation  Xchange,  InnoCentive) 

•  Functions  as  an  honest  broker 

Helps  PMs  locate  suitable  M&S  resources 
Helps  developers  find  a  market  for  established  M&S  resources 
Independent  of  developers  and  users  -  Free  to  sign  NDAs 
Documents  legal  status  of  each  M&S  resource  within  DoD 

•  Facilitates  license  agreements 
Manages  tiers  of  licenses  across  DoD 
Builds  and  maintains  the  knowledge  base 

How  resources  have  been  used  in  the  past 
-  V&V  histories 

Handles  MOUs  to  guide  appropriate  use  and  avoid  liability 


Virtual  collaboration  through  electronic  registries  alone 
will  be  insufficient  to  achieve  desired  levels  of  reuse 
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Business  model  actions  that 

will  spur  M&S  reuse  (5  of  7) 

•  Establish  enablers  for  open  business 
model  transactions  for  both  government 
and  industry 

-  Register  reusable  M&S  assets  (Gov’t  and 
industry) 

Include  license  rights 

Include  info  on  previous  applications 

-  Allow  user-wiki  comments  on  experiences  with 
the  M&S 


CNA 
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Business  model  actions  that 
will  spur  M&S  reuse  (6  of  7) 


•  Explore  the  transfer  of  reusable  M&S 
resources  to  a  working  capital  fund  (e.g., 
major  test  range) 

-  Compensate  M&S  provider  with  test  range 
services 

•  Develop  methods  to  assess  downstream 
and  cross-program  reuse  potential 

•  Adopt  strong  scientific  practices  to 
ensure  credibility  of  M&S  products 


CNA 
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Business  model  actions  that 

will  spur  M&S  reuse  (7  of  7) 

*  Promote  the  use  of  open  source  software 

*  Grant  industry  access  to  approved  government 
models  and  databases 

*  Add  reuse  as  performance  objective  for  Gov’t 
stewards  of  M&S  funds 

-  Examine  registry/repository  first 

Fund  to  make  new  M&S  reusable  for  others 

*  Pursue  balanced  acquisition  strategy 

-  M&S  COTS  with  tier-based  licenses,  GOTS,  GPR,  and 
proprietary  non-commercial  where  needed 

*  Publicize  DoD  M&S  reuse  objectives  and 
strategy) 

-  Use  keynote  address  at  conferences/  articles  in  trade 
journals  and  professional  societies 


CNA 
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Integrating  Architecting  and 
Systems  Engineering 


NDIA  SE  Conference 
22  October  2008 
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Intro/Topics 


Background:  What  is  the  Problem? 

Why  is  An  Architecture  Framework  Needed? 


Organizations  are  developing  major  systems  that 
need  to  interface  and  interact 


CINCs 


USACOM 


♦  Q  4  4 


Services 


Q  4  Q  Q 


Agencies 


DLA  NIMA  DISA 

«  «  » 


Differences  in  content  and  formats  inhibit  comparison  of 

architectures 


Disparate  and  unrelatable  architecture  products  lead  to 
non-integrated,  non-interoperable,  and  non-cost  effective 

capabilities  in  the  field 


Reprinted  from  “C4ISR  INCOSE  Tutorial”,  A.H.  Levis  and  L.W.Wagenhals,  March  2001 
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Evolution  DoDAF 


1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009? 


DoDAF  v.  1.5 


Clinger- 

Cohen 

C4ISRAF  v.  2.0 

Improvements 


A 

/  \ 


Draft  DoDAF  v.  2.1 
Object-oriented  methodology 


C4ISRAF  v.  1.0 

New  Standard 
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MBSE5  -  4 


Motivations  for  DoDAF 


Architectures  required  by  law  (Clinger-Cohen, 
etc.) 

Structured,  repeatable  method  for  investments 
and  investment  alternatives 

Influence  and  guide  organizational  change 

Create  New  Systems  ( i.e define  System 
Requirements) 

Deploy  (plan  for)  new  technologies 
-  Ex.,  Net-Centric  Warfare 
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Typical  DoDAF  Taxonomy 


Operational  Concept 


OV-1 


TTP 


OV-4 


-DRM 

OpSits 

- 

OV-5 


DRM: 

Design  Reference  Mission 

OpSit: 

Operational  Situation 

TTP: 

Tactics,  Techniques,  Procedures 

FoS: 

Family  of  Systems 

SoS: 

System  of  Systems 

Functional  Mapping 


SV-3“  - - 

SV-4 


OV-3 


SV-5 


1st  Order  Analysis: 
Functionality 


Interface  Mapping 


OV-2 


-  - 

-i  J  iBl-- 

SV-1  TV-1 


OV-3  SV-2"  SV-6 


Note:  There  are  dependencies  between  the 

Architecture  products  that  are  not  shown  in 
the  System  Engineering  flow.  Many  of  the 
products  are  developed  concurrently. 


Architectures  Provide  the  Framework  for  FoS/SoS 
Systems  Engineering  &  Acquisition 


OV-1 

High-level  Operational  Concept  Graphic 

OV-2 

Operational  Node  Connectivity  Description 

OV-3 

Operational  Information  Exchange  Matrix 

OV-4 

Command  Relationships  Chart 

OV-5 

Activity  Model 

OV-6C 

Operational  Event/Trace  Description 

SV-1 

System  Interface  Description 

SV-2 

Systems  Communication  Description 

SV-3 

Systems  Matrix 

SV-4 

System  Functionality  Description 

SV-5 

Operational  Activity  to  System  Function 

Traceability  Matrix 

SV-6 

System  Information  Exchange  Matrix 

SV-7 

System  Performance  Parameters  Matrix 

SV-8 

System  Evolution  Description 

SV-9 

System  Technology  Forecast 

SV-10 

System  Activity  Sequence  &  Timing 

TV-1 

Technical  Architecture  Profile 

TV-2 

Standards  Technology  Forecast 

FoS/SoS  Evolution 

Static  Interoperability 


Ref:  “Naval  Collaborative  Environment”,  Dr.  Harry  Crisp,  2002 


Architecture  Performance 
and  Behavior 


OV-6C  3 


SV-7 


SV-10 


Executable 
Model 


3rd  Order  Analysis: 
Dynamic  Interoperability 
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Integrated  Architectures  -  Defined 


Architecture  data  elements  uniquely  defined  and 
consistently  used 

Accomplished  through  the  mapping  of 
standardized  terms,  definitions,  and  relations 

-  Objects  used  in  more  than  one  view  are  identical 

-  Objects  linked  between  views  are  linked  within  an 
underlying  data  base. 

Common  points  of  reference  linking  different 
views  of  the  architecture 

Examples 
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View  Creation  Not  Complete  System 

Architecture 


No  Requirements 

Need  for  integration  with  other  SE  related 
activities  -  (Test  Planning) 

Representations  of  Traceability  lacking 
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Review  of  MBSE 


•  Model-driven  approach  to  capture  and 
integrate: 

-  Requirements  Development 

-  Logical  Analysis 

-  Design  Solution 

-  Implementation 

-  Integration 

-  Verification 

-  Validation 

•  System  Specification  is  the  model,  Model 
is  the  System  Specification 
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Example  MBSE  Taxonomy 


Source  Requirements  Domain 


Behavior  Domain 


Originating  requirements 
trace  to  physical  components 
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Example  MBSE  taxonomy  (cont.) 


Source 

Documents 


1 

_ _ ► 

Reqts. 


Next-level 

Reqts. 


Next-level 

Reqts. 


Primary  Concurrent  Engineering  Activities  At  Each  Layer 


Originating 

Reguirements 

Analysis 


Behavior 

Analysis 


Synthesis/ 

Architecture 


Design 
V& V 


System  Design  Repository 

Specification  &  Report  Generation 

Iterate  as  required 

When  layer  completed 

i=>  Layer  1 
(Draft  1) 


Initial  Requirements  for 
this  layer  are  embodied 
in  the  model  passed 
from  the  prior  layer 


Behavior 

Analysis 

c=i-C0  >1=11=1 


Synthesis/ 

Architecture 


Design 
V& V 


c^Layer  2 
(Draft  2) 


System  Design  Repository 


Specification  &  Report  Generation 


ft 


Iterate  as  required 


When  layer  completed 


Initial  Requirements  for 
this  layer  are  embodied 
in  the  model  passed 
from  the  prior  layer 


Behavior 

Analysis 


Synthesis/ 

Architecture 


Design 
V& V 


System  Design  Repository 


Specification  &  Report  Generation 


Layer  n 

(Final 

Specs.) 


Must  complete  a  layer  before  moving  to  the  next  layer  (completeness) 

.  "  "  5) 


Cannot  iterate  back  more  than  one  layer  (convergence] 
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Vitech 


MBSE  and  Integrated 
Architecture  Common  Traits 
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DoDAF  Integrated  Data  Layer 


Architecture  Framework  Structure 


Presentation  Layer 


Synchronized 


Products  Views 


Data  Layer 

Architecture  Data  Elements 


|  [«>  )*■(  ITf-JITWIMl 


■  r’rM  1 

Pr  r  F.7J  niuu 


^  rt  r  r-M  nwn.-r  ] 

* 


[DoDAF  1.5]  13 
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071015 


MBSE  Integrated  Data  Layer 


Requirements 

Management 


EZ - — I- — — L 


Behavioral 


Architecture 

Synthesis 


Verification 


Design 

Specifications 


Integrated,  Consistent  Analysis:  Complete  Specifications,  Project  Documentation, 

Queries  &  Models 
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[DoDAF0IZ5piS 


Integrated  DoDAF  Data  Model 


Operational 

System 

Architecture 

Architecture 

Domain 

com  Dosed  of _ 

Architecture 

rtf 

Domain 

built  from 


Operational 

/I  implemented  by 

Comoonent 

Node 

J  S  1  ipui  ICl  1 1 

built  from 


performs 


connected  to 
J  thru 


achieves) 


Interface 


Guidance 


guides 


NeedLine 


exhibits. 


comprised  of 


implemented  b 

T 


joins  & 
joins 


Link 


includes 


Exchange 

Characteristic 


Selected 

Classes 


connected 
to  /  thru 

performs 


transfers 


transfers 


exhibits 


Operational 

Information 


decomposed  by 


I 
I 

/implemented  bv 

I  * 


Exchange 

Characteristic 


exhibits 


Item  ^  decomposed  by 


inputs  / 
outputs  / 


decomposed  by 


Operational 

Activity 

ouipuis  /  1 

triggered  by  / 

H  implemented  by 

'  > 

Pi  in/ 

f 

'tinn 

- - - 1  - - - 

Operational 

Task 


captures  / 
consumes  / 
reduces 


captures  / 
consumes  / 
produces 


Mission 


includes 


includes 


Organization 

T 


decomposed  by 


Color  Code 


includes 
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Physical 

Element 

Interface 

Element 

J  Functional 
!  Element 

Requirement 

Element 

Systems  Engineering  Data  Model 

(partial) 


Source 

Documents 


Requirement 

(Performance) 


Requirement 

(Functional) 


Requirement 

(Constraint) 
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Integrated  Data  Model:  Complete  Traceability  Between  the 
Operational  Architecture  and  System  Engineering  Domains 


Operational 

System 

Architecture 

Architecture 

Domain 

composed  of _ 

Architecture 

rtf 

Domain 

built  from 


Operational 

/I  implemented  by 

Comoonent 

Node 

UUII  ipui  ICl  1 1 

performs 


connected  to 
J  thru 


achieves) 


Interface 


Guidance 


NeedLine 


exhibits. 


comprised  of 


implemented  b 

T 


joins  & 
joins  thru 


Link 


guides 


includes 


Exchange 

Characteristic 


Selected 

Classes 


transfers 


exhibits 


Z'5*  Operational 
Information 


I 

I 


transfers 


decomposed  by 


/implemented  bv 


connected 
to  / thru 


c 


built  from 


Standard 

3”  ■  ■ 


includes 


xhibits  performs 


ns 


Exchange 

Characteristic 


Selected 

Classes 


Requirement 


Item 


exhibits 
jcomposed  by 


Operational 

Activity 


inputs  / 
outr 


inputs  / 
outputs  / 
.triggered  by 


implemented  by 


decomposed  by 


Operational 

Task 


'  ^.exits  by^  - - ^ 

Zd 

Exit 

^  exits  by  ^ 

captures / 

captures  / 

refined, 
basis. 


specifies 


Function 


Selected 

Classes 


consumes  / 
reduces 


Resource 


consumes  / 
produces 


Mission 


Organization 


decomposed  by 


Color  Code 


includes 


includes 


includes 


responsible  for 


Selected  Classes 


1995-2006  Vitech  Corporation.  All  rights  reserved. 


Physical 

Element 

Interface 

Element 

J  Functional 
!  Element 

Requirement 

Element 

Vitech 


DoDAF  and  MBSE  System 
Model  Overlap  -  Examples 
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Sample  Project 
Tactical  Imagery  Gathering 


Description:  The  Tactical  Image  Management  Architecture  is 
composed  of  both  an  operational  element  and  an  image 
management  system  which  supports  the  architecture. 

The  tactical  scenario  models  an  army  platoon  which  is 
advancing  over  a  hill  and  requires  information  about  the  tactical 
environment  on  the  other  side  of  hill.  The  platoon  makes  an 
image  information  request  which  is  transferred  back  to  joint 
task  force.  The  joint  task  force  has  access  to  an  image 
management  system  which  checks  to  see  if  the  information 
required  is  already  available  in  its  inventory.  If  the  infgpmrttbn 
is  not  in  the  inventory,  a  tactical  UAV,  in  thisj&se  a  Predator,  is 
tasked  to  collect  an  image  of  the  otbertfcie of  the  hill,  send  it 
back  to  the  image  managpmefifsystem,  and  then  the  requested 
tactical  informatierTfscommunicated  to  the  platoon. 


OV-1.  Scenario  1 


Wide  Area  ^ 
Surveillanc^^^ 


Scenario  1:  Tactical  UAV 
Controlled  by  JTF  HQ  (OV-1) 
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Architectures 
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Modeled  Relationships 
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Architecture  Traced  to  Guidance 

Documents 
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As  seen  in  the  produced  AV-2 


ARCHITECTURE  DESCRIPTION  DOCUMENT 
FOR 

Tactical  Image  Management 


Wednesday,  October  15. 2008 


Prepared  For: 

Tutorial  Participants 
1441  QuiveraRd 
SsaDies>.CA92109 


Prepared  By 


NDIA  Presenter 
123  Main  Street 
Spring  Lake,  NJ  08'3« 


Joint  v  isionJi FIU 

Includes  Subordinate  Guidance: 

JV.3 . 5  Implications  of  Technological  Advances 

Guides: 

Operational!^ ode:  IM  TACIM  Op  Scenario  Participants 

lg  Importance  of  Information  Superiority' 

Description: 

We  must  have  information  superiority:  the  capability  to  collect,  process,  and  disseminate  an " 
uninterrupted  flow  of  information  while  exploiting  or  denying  an  adversary  s  ability  to  do  the/ame. 


Strateg 


Source  Document!  s  ): 

Jomt  Vision  2010 

TnHnH^HTn  gad  GinHance 

Tj  implications  of  T echnologicaHT 
Guides: 

Architecture:  Mod  Tactical  linage  Management 


JY.4  Conduct  of  Joint  Operations 

.toOL. _ 

Pages !  “  tnrougn  ^  i  oi  j  v  j jIIT 

S  ource  Documents): 

Joint  Vision  2010 

Includes  Subordmate  Guidance: 

JV.4. 1  New  Operational  C oncepts 
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Architecture 
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Architectures  -  Example 
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Capability  -  Support  External  Users 


IMS 


— >{and) 


Not  Authorized 


C.2.1 


Validate 

Authorized  User 


c2.User 

jthorfzat 


External  Users 


Date: 

Thursday,  July  26  ,  2007 


Number: 


C.2 


Author: 


C.2.3 

C.2. 4 

Authorized  User  „ 

Accept  Search 
Request 

Provide  search 
results 

w 

w 

Vitech  Corporation 


Name: 

Support  external  uses  of  IMS  inventory  products 
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Support  External  Users  as  an  OV-5 


T 


Support  external  uses  of  IMS  inventory  products 
Operational  Activity  Model  (OV-5) 


"AQM  Oo  Scensno  92<”3eM,vSr'.ene orf  S tzravtn 


C2  Support  external  uses  of  IMS  inventory  products  IDEFO  Diagram 
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Architectures  &  Domains 
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One  View 


Hierarchy  of  Operational  Nodes 


SOA 

Service  Oriente. . . 

OperationalNode 

Context 


[built  from  |buiit  from 


EC 

IM 

External  Custo... 

TACIM  Op  Scena... 

OperationalNode 

OperationalNode 

External 

Operational  Arch... 

|built  from 

|built  from 

[built  from 

|built  from 

built  from 

|built  from 

|built  from 

EC.l 

EC. 2 

IM.  1 

IM.2 

IM.3 

IM.4 

IM.5 

Inventory  Sear... 

New  Product  Cu... 

JTF  Level  (Land) 

Brigade  Level 

Platoon  Level 

Image  Manage... 

TUAV 

OperationalNode 

OperationalNode 

OperationalNode 

OperationalNode 

OperationalNode 

OperationalNode 

OperationalNode 

Organizational  R... 

Organizational  R... 

Operational  Ele. . . 

Operational  Ele... 

Operational  Ele... 

Operational  Ele... 

External 

Date: 

Thursday,  July  26,  2007 

Author: 

Vitech  Corporation 

Number: 

SOA 

Name: 

Service  Oriented  Architecture  Context 
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Relates  to  the  Next . . . 


I/O  of  Operational  Activities 


Date: 

Thursday,  July  26,  2007 

Author: 

Vitech  Corporation 

Number: 

C.2 

Name: 

Support  external  uses  of  IMS  inventory  products 

Date: 

Author: 

Thursday,  July  26,  2007 

Vitech  Corporation 

Number: 

Name: 

C.2 

Support  external  uses  of  IMS  inventory  products 
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. . .  Relates  to  the  Next . . . 


TACIM  Op  Scenario  Participants 
Operational  Node  Connectivity  (OV-2) 


i  '•  ’ : 


Ose- 


T~W 

i 

- 

— 

£ 


ar  ^sct  Levtf 


r  e-~-g  ~ 1 


^<3CS! 


Votoe  GBH-r-  .meteors 


O^  r.c-v  £er&-/ 


<5 


2JM  CCm-JTP  CGrrxrYcMors 


IM4 


l-vei«  MBmgerwc 
Cft- 


tent' 

- * - 


1M  5 


Innenpry  acr  *o»s  ^ 


^ftewp 


prccjfltser- 


35 


Ece~o  Cjflton>efs 


iix 

WeOKScay.  Oocoer  u  2C0i 

AuT**: 

TtaTHttm 

S-r-ce-: 

1M 

rtsr-e: 

TAZLM  Co  Soensrio  flsrtwosns 

. . .  And  From  Our  Integrated 

Architecture . . . 
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. . .  The  Resulting  OV-3 


Tactical  Image  Management 

Operational  Information  Exchange  Matrix  (OV-3) 

PART  I 

Need  line 

Information 

Exchange 

Operational  Information 
Element 

Information  Source 

Information  Destination 

Brigade- JTF 
Communications 

Brigade  -  JTF  Exchange 
Characteristic 

cl. Collected  Information 

Description:  Package  of 
imaging  information  and 
augmenting  material  returned  to 
the  customer. 

Accuracy:  Medium 

JTF  Level  (Land) 

Transmit  Co  Use  ted 

Itfo  rmationCiS’Is ) 

Brigade  Level 

Translate  Information  into 
Verbal  CommandsCis-b) 

cl. Formatted  RFI 

Description:  FormattedRequest 
for  Informationreque  sting 
intelligence  on  a  target  at  a 

Brigade  Leva! 

Req  uest  Latest  1 tformatio  n 
for  Location  (As-h) 

JTF  Level  (Land) 

Rsc  ehe  Fo  rmatxed 
RFl(As-Is) 

Accuracy:  High 

> 

iMCell-JTF 

Communications 

1M  Cell  -  JTF  Exchange 
Characteristics 

cl  .Tactical  Image  Products 

Description:  Processed  imaging 
products. 

Accuracy:  Medium 

Imase  Management  Cell 

Provide  Current  Target 
Imaging  ProductCis-b) 

JTF  L  evel  (Land) 

Process  Tactical 

c  1  .Tactical  T asking  Imaging 
Request 

JTF  Level  (Land) 

1  AMfl  fl  f/1  . .  ^ 

Image  Managemgjtfiei!^^ 

r  Tn  zr  Mjr  i  flfitif  tA 

"Description:  lasting  re-quest  to 
acquire  imaging  intelligence  for 
tactical  commanders. 

Accuracy:  High 

ivCvCiVC  i  4/  / 

IWlfAs-ls) 

jAvv  Slri  X  Ivf  i  iiw  Uv  Ui 

Operanons(As-Is) 

IMCell-TUAV 

Communications 

c  1  .Target  Imaging 

TUAV 

TUAV  Ops(As-ls) 

Image  Management  Cell 

Acquire  Target 
ImagingCis-k) 

Image  Management  Cell 
Subscribers)  for  th  is 
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Architect 


Operational  Architecture  Domain 

Operational  „need" 

[read:  Capability] 

translation 

process 


* 


Requirements 
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&  Requirements 


translation 

process 


Systems  Architecture  Domain 


Design 


reserved. 
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Capability  to  Requirement  Traceability 


©  1995-2006  Vitech  Corporation.  All  rights  reserved. 


MBSE5  -  34 


Requirement  to  Function  Traceability 
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Op  Activities  implemented  by 

System  Functions 
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SV-5a  Operational  Activity  to  Systems  Function 

Traceability  Matrix 


Support  external  uses  of  IMS  inventory  products  to 

Operational  Activity  to  Systems  Function  Traceability  Matrix  (SV-5) 

Function 

Operational  Activity 

Accept  Search  RcqucHl 

o 

g 
•  = 

s 

c 

w 

.2 

c 

ac 

d 

o 

cc 

b 

f 

"3 

— 

o 

* 
u : 
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■ 

B 

ac 

i 

-a 

> 

| 

W 
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ac 
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“C 

8 

•p 

p 

c 

J2 

i 

> 

Authorize  User 

> 

X 

Check  Subscriber  Requests 

Determine  Sensor  Mix 

Distribute  New  Product 

Existing  Sub  scriptions  ? 

Fly  to  Surv  eillance  Po  s  iticm 

Get  Product  From  Inventory 

Get  Search  Parameters 

X 

Get  Subscription  Parameters 

New  Product  Received 

Not  Authorized 

X 

X 

Perform  Inventory  Search 

X 

Perform  Predator  Surveillance 

Prioritize  Request 

—  _  —  Vvednesdav  October  2006 
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SE  Traceability 


|basis  of 


basis  of 


performed  by 


S.  1.4 


Workstations 


Component 


Subsystem 


OR.  1 


COLLMGT.txt 


Document 


System  Requir.. 


[documents 


documents 


S.l 

OR.  1 

Image  Manage... 

General  Requir... 

Component 

Requirement 

Systems  Archite... 

Comp 

osite 

refined  by 

|refined  by 

refined  by 

refined  by 

refined  by 

OR.  1.1 

^OR.1.2 

^OR.1.3 

IOR.1.4 

t»R.1.5 

Accept  Requests 

Retain  Inventory 

Control  Multiple... 

Prioritize  Reque... 

Monitor  and  Ass... 

Requirement 

Requirement 

Requirement 

Requirement 

Requirement 

Functional 

Functional 

Functional 

Functional 

Composite 

basis  or 


IMS.  1 

IMS.  1 

IMS.  17 

Accept  and  For... 

Existing:  Accept... 

Authorize  User 

Function 

Function 

Function 

Date: 

Author: 

Friday,  August  25,  2006 

Vitech  Corporation 

Number: 

Name: 

OR.  1 

COLLMGT.txt 
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Function  Appears  in  the  SSS 
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As  Part  Of  SV-4 


renorni  CoUecfluo  MaDigeintni 
SvsifmvStrvtres  fnnc’tJnaaliry  I)t  irnptlon  (SY  4) 


l— £3- 


IMS  Prrfarm  C  glkrtv*  Uaucrani 


IMS.18 

— ET* 

Get  Subscription 
Parameters 

IMS.  19 


Register 

Subscription 

Information 


/  search  \ 

IMS.  21 

— cr* 

Get  Search 

Parameters 

IMS. 22 


Perform 
Inventor/  Seard 


f  (  *Lj) 

V  actnoroecy 

•* 

IMS. 24 

Not  Authorise 

Date: 

Wednesday,  October  15.  200 

Author 

l  Tim  Tntsch 

Number: 

IMS 

Name: 

pgig;.r,  ufljsaar.  Kaaas.T.a; 

Management  Enhanced  FFBD 
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Inside  the  Data  Model 
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N2  Diagram  Provides  a  Snapshot  of 

System  I/O 


Date: 

Monday,  June  25  ,  2007 


Number: 


IS,  3 


Author: 


Vitech  Corporation 


Name: 


Subscription  Service  Root 
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Analyst.  Ifttfd... 


Functional  Allocation  to  System  Components 
_ Reveals  Required  Connectivity 


System  Interoperability  Also  Used  In 
Interface  Requirement  Specifications 
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SV-3  Systems -Systems  Matrix 


Image  Management  System 

Systems-Systems  Matrix  (SV-3) 


m 


Analysts 

Comm  and  Center 

Ground  C  ontrol  Station  (GCS[) 
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Predator  Vehicle 

Work  Stations 

Tactical  Customers 

Analysts 

X 

Command  Center 

X 

X 

X 

Ground  Control  Station  (GCS) 

X 

X 

X 

Predator  Crew 

X 

Predator  Vehicle 

X 

Workstations 

X 

X 

Tactical  Customers 

X 
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Test  Planning  In 


3.2.1  Accept  Request  Test 
a)  SCHEDULE: 

Estimated  Duration  Start  Date 


End  Date 


b)  TEST  CONFIGURATION 

The  Accept  Request  configuration  consists  of  one  or  more  tactical  customers  conn« 
and  entering  new  product  requests. 


Test  Equipment 

System  Context 


Description 


TEST  AND  EVALUATION  PLAN 
FOR  THE 

I\LAGE  MANAGEMENT  SYSTEM  WITH  SERVICES 


Tutorial  Participant! 

1441  QnivaaRd 
SanDi»§B,CA92109 


A  reference  component  used  to  incorporate 
and  a  system  under  one  physical  represents 


c)  TEST  TEAM: 

d)  TEST  PROCEDURES: 

1.  TP.  Accept  Request  Test  steps  are  documented  here: 

1 .  View  active  job  log 

a.  Document  existing  IMS  jobs  in  progress 

2 .  Enter  new  product  request 

3.  Ensure  product  request  is  entered  into  IMS 

3.1  View  active  job  log 

3 .2  Document  new  IMS  job  has  been  entered  into  active  list 


NDIA  Presenter 
123  Main  Street 
Spring  Late.  XJ  08734 


MS.  1 


Accept  and  Format  Request 


Function 


verified  by 


M.3 


Accept  and  Format 


VerificationRequirement 


[fulfilled  by 


jfijIfiHed  by 


Accept  Request  Test 

Format  Request  Test 

VerificationEvent 

VerificationEvent 

|employs 

|employs 

TC.  Accept  Request 

TP.  Accept  Request 

TestConfiguration 

TestProcedure 

Date: 

Sunday,  October  19,  2008 


Author: 


Number: 


IMS.  1 


Vitech  Corporation 


Name: 


Accept  and  Format  Request 
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Architecture  Executeability: 
Operational  and  System  Level 


w 


Timeline  Elements  for  "Content  Function  -  Level  2"  (Default) 


File  Results  Options  Help 


a  ■  ou 


Execution  Hierarchy 

■0SHZZ3BHT 


O  1  Parallel 

O  1 .1  Branch  User 
L . •1.1.1  Parallel 

•  1.1. 1.1  Branch 

O  m  1.1.111  0.1.1  Make  Information  Request 
•G  1.1. 1.2  Branch 

On  1.1. 1.2.1  0.1 .2  Receive  Estimated  Schedule 
Q  1.1. 1.3  Branch 

i . •  [T]  1.1. 1.3.1  0.1 .3  Accept  Products 

O  1 .2  Branch  System 

0|T|  1.2.1  1  Accept  And  Format  Request 
On  1.2.2  2  Check  Product  Inventory 
O  1 .2.3  Exit  From  2  Check  Product  Inventory 
0  1.2.3.1  Branch  In  Inventory 

•  1.2. 3. 2  Branch  Not  In  Inventory 

•  1 .2. 3. 2.1  3  Prioritize  Request 

•  1 .2. 3. 2. 2  4  Determine  Collector  Mix 

•  1.2. 3. 2. 3  Parallel 
-  •1.2. 3. 2. 3.1  Branch 

i . •  1.2. 3. 2. 3. 1.1  5  NotilV  User  Of  Estirr 

L ...*  1.2. 3. 2. 3. 2  Branch 

L  •  1.2. 3. 2. 3. 2.1  6  Task  Collectors 

•  1 .2. 3. 2. 4  7  Accept  And  Format  Collector  P 

•  1 .2. 3. 2. 5  8  Put  Product  In  Inventory 
On  1.2.4  9  Get  Product  From  Inventory 
O  1.2.5  Parallel 

0  1.2.5.1  Branch 

i . On  1 .2.5.1 .1  1 0  Provide  ProductTo  User 

O  1.2. 5. 2  Branch 

On  1.2.5. 2.1  11  Evaluate  Products  vs.  Ret 
■0  1.2.5.2.2  Exit  From  11  Evaluate  Products ' 


5.1.1  Process  Sensor  Returns 

5.1.2  Update  Track  File  1 

5.1.3  Update  TC  Display 

5.1.4  Issue  Tower  Handoff 

5.2  Tower  Control 

5.3  Land  Plane 

5.4  Taxi  to  Gate 


Transcript 


Expand  All 


28 1 . 0348 1 8,  ID(4),  PR0C(2),  finish,  1 ,  Parallel 

281 . 034818, (aux),PROC(0), enabled, 2„5.3,"Land  Plane" 

28 1 . 0348 1 8,  (aux),  PROC(O),  waitingForResources,  2, ,  5 . 3,  "Land  Plane" 
281 .034818, (aux), PROC(O), dequeued, 2, ,5. 3,  "Land  Plane", "Landing  Cc 
28 1 . 0348 1 8,  ID(24 1 ),  PROC(O),  start,  2„  5 . 3,  "Land  Plane" 

284 .03481 8,  ID(242),  PROC(O),  finish,  2„  5 . 3,  "Land  Plane" 

284. 034818, (aux), PROC(O), enabled, 3„5. 4, "Taxi  to  Gate" 

284.034818, (aux), PROC(O),  waitingForResources,  3„  5 . 4,  "Taxi  to  Gate" 
284. 034818, ID(243),PROC(0), start, 3„5.4, "Taxi  to  Gate" 

289. 034818, ID(244),PROC(0), finish, 3„5. 4, "Taxi  to  Gate" 


System  behavior  diagram  defines 
architecture  of  simulation  model  which 
allows  us  to  analyze  Behavior,  Timelines, 
Resources,  Queues,  and  Flows 


j 

if1 
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Integrated  Model  Benefits 


Synchronization  between  DoDAF  views  and 
systems  Engineering  products 

Traceability  of  Operational  Doctrine  to  System- 
level  Functional  Requirements 

-  Can  be  establish  through  Operational  Scenarios 

-  Supports  Operational  Testing 
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Discussion 


•  Implications  for  CADM  (?) 

•  DoDAF  Views  lend  themselves  to  analysis,  not 
system  development 

•  Migration  from  C4ISR  to  DoDAF 

-  Typical  modeling  techniques  limited  to  computer  stuff 

-  Much  discovery  work  goes  straight  to  software 

-  Traced  from  TOGAF? 
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Things  to  Mention 


•  TOGAF  -  The  Open  Group  Architecture 
Framework 

-  TOGAF  ADM  -  Architecture  Development  Method, 
limited  to  amorphous,  distributed  computer  gunk 

•  SE  principles  applicable  to  all  levels  of  analysis 

•  Why  is  the  DoD  “Chief  Information  Officer” 
dictating  methods  and  tools  by  which  we 
develop  systems? 

•  “Interoperability”  issues  not  limited  to  “purple” 

-  Including  “disadvantaged”  or  “tactical  edge”  users  (the 
real  war  fighters) 
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The  End 
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Systems  Engineering 

for 

Systems  of  Systems 


NDIA 

SE  Conference 
October  2008 

Dr.  Judith  Dahmann 
The  MITRE  Corporation 


SoS  SE  Challenge 

•  US  DoD  builds  and  fields  large  systems  employed  to  support 
Joint  and  Coalition  operations 

-  Conceived  and  developed  independent  by  Military  Services 

-  Acquisition  (and  SE)  on  a  system  by  system  basis 

•  Focus  of  DoD  investment  shifting  to  broad  user  capabilities 
implemented  in  a  networked  environment 

-  Mix  of  material  and  non-material  assets  which  must  work  together  to  meet 
capability  objectives 

-  Individual  systems  are  no  longer  considered  as  individual  bounded  entities 
and  are  evolved  based  on  extant  capabilities 

-  Components  in  larger,  more  variable,  ensembles  of  interdependent  systems 
which  interact  based  on  end-to-end  business  processes  and  networked 
information  exchange 

•  Increasingly  SoS  of  various  types  proliferate  despite 
continued  focus  on  individual  systems 


What  are  the  implications  for  SE? 


Version  1 .0  I  SoS  Guide  Version 


DoD  System  of  Systems  SE  Guide 


•  Effort  led  by  the  Office  of  the  Secretary  of  Defense 

•  Collaborative  Approach  with  DoD,  Industry,  Academia 

•  Purpose 

-  6  month  effort  addressing  areas  of  agreement  across  the  community 

-  Focus  on  technical  aspects  of  SE  applicable  across  SoS  management  constructs 

-  Vehicle  to  capture  and  debate  current  SoS  experience 

•  Audience 

-  SoS  and  Program  Managers  and  Lead/Chief  Engineers 

•  Develop  ‘Boots  on  the  Ground'  basis  for  Version  1 .0 

-  Structured  reviews  with  practitioners 

-  Refine  early  draft  guide  content,  identify  areas  for  future  study 

•  Update  findings  and  release  Version  1.0 

-  Draft  released  for  comment  December  2007 

-  -600  comments  received  in  February  2008  (Industry,  FFRDCs,  Gov’t) 

-  Revision  reviewed  by  Senior  SE  leadership  in  July  2008 

-  Final  release  in  August  2008 


What  does  SoS  Look  Like  in  the  DoD  Today? 


•  Typically  an  overlay  to  ensemble  of  individual  systems 

brought  together  to  satisfy  user  capability  needs 

•  Are  not  new  acquisitions  per  se 

-  Cases  like  FCS  are  extremely  rare  and,  in  practice,  still  must 
integrate  with  legacy  systems 

•  SoS  ‘manager’  does  not  control  the  requirements  or 
funding  for  the  individual  systems 

-  May  be  in  a  role  of  influencing  rather  than  directing,  impacts  SE 
approach 

•  Focus  of  SoS  is  on  evolution  of  capability  over  time 

•  A  functioning  SoS  takes  start-up  time  but,  in  steady 
state,  seems  well-suited  to  routine  incremental 

updates 


Most  military  systems  are  part  of  an  SoS  operationally 
Only  by  exception  do  we  manage  and  engineer  at  SoS  level 


Definitions 


SoS:  A  set  or  arrangement  of  systems  that  results  when  independent  and  useful 
systems  are  integrated  into  a  larger  system  that  delivers  unique  capabilities 
[DoD,  2004(1)]. 

Accepted  Taxonomy  of  SoS  [Maier,  M.  1998] 

-  Directed 

•  SoS  objectives,  management,  funding  and  authority;  systems  are 
subordinated  to  SoS 

-  Collaborative 

•  No  objectives,  management,  authority,  responsibility,  or  funding  at 
the  SoS  level;  Systems  voluntarily  work  together  to  address  shared  or 
common  interest 

-  Virtual 

•  Like  collaborative,  but  systems  don't  know  about  each  other 

US  DoD  Pilots  identify  a  new  SoS  type: 

-  Acknowledged 

•  SoS  objectives,  management,  funding  and  authority;  however  systems 
retain  their  own  management,  funding  and  authority  in  parallel  with 
the  SoS 


SoS  SE  Guidebook  focuses  on  ‘Acknowledged’  SoS 


Characteristics  of  Acknowledged  SoS 

Top-down  direction  for  an  SoS  capability  concurrent  with 

independent  direction  and  autonomy  in  system  operation 
and  development 

-  Multiple  levels  of  objectives 

-  Multiple  management  authorities  with  independent  priorities, 
funding  and  development  plans 

-  Multiple  technical  authorities 

Much  of  SoS  functionality  is  in  extant  capabilities  of  the 
systems 

SoS  manager  and  SE  do  not  have  control  over  all  the 
parts  of  the  SoS 

-  In  fact,  they  may  not  be  aware  of  all  the  systems  which  may 
impact  their  objectives  and  both  the  systems  and  the  objectives 
may  change  over  time. 


Management  of  Acknowledged  SoS 

•  Independent,  concurrent  management  and  funding 
authority  pose  management  issues 

•  In  defense,  a  solid  governance  &  management  approach  is 
seen  as  key  for  SoS 

-  Independent  authorities  are  unlikely  to  accept  direction  from  a  systems 
engineer  they  do  not  control 

-  Argue  to  make  'acknowledged'  into  'directed'  made  difficult  by  'multi- 
mission'  systems  which  are  important  to  multiple  SoS 

•  Beyond  defense  ‘acknowledged’  SoS  exist  and  evolve 
without  top  down  management 

-  Systems  or  services  are  designed  to  be  broadly  useful  and  have  as  their 
business  objective  to  support  numerous  user  applications 

-  They  naturally  retain  authority  over  decisions  regarding  their  development 
and  are  not  likely  to  agree  to  limit  themselves  to  one  specific  customer 


Management  issues  have  technical  implications  for  SE 


A  Comparison 


System 

System  of  Systems 

Management  &  Oversight 

Stakeholder 

Involvement 

Clearer  set  of 
stakeholders 

Two  levels  of  stakeholders  with  mixed  possibly 
competing  interests 

Governance 

Aligned  PM  and  funding 

Added  levels  of  complexity  due  to  management  and 
funding  for  both  SoS  and  systems;  No  SoS  does  over  all 
systems 

Operational  Environment 

Operational 

Focus 

Designed  and  developed 
to  meet  operational 
objectives 

Called  upon  to  meet  operational  objectives  using 
systems  whose  objectives  may  or  may  not  align  with 
the  SoS  system’s  objectives 

Implementation 

Acquisition 

Aligned  to  established 
acquisition  processes 

Cross  multiple  system  lifecycles  across  acquisition 
programs,  involving  legacy  systems,  developmental 
systems,  and  technology  insertion;  Capability 
objectives  but  may  not  have  formal  requirements 

Test  & 
Evaluation 

Test  and  evaluation  the 
system  is  possible 

Testing  more  challenging  due  systems’  asynchronous 
life  cycles  and  given  the  complexity  of  all  the  moving 
parts 

Engineering  &  Design  Considerations 

Boundaries 
&  Interfaces 

Focuses  on  boundaries 
and  interfaces 

Focus  on  identifying  systems  contributing  to  SoS 
objectives  and  enabling  the  flow  of  data,  control  and 
functionality  across  the  SoS  while  balancing  needs  of 
the  systems 

Performance 
&  Behavior 

Performance  of  the 
system  to  meet 
performance  objectives 

Performance  across  the  SoS  that  satisfies  SoS  user 
capability  needs  while  balancing  needs  of  the  systems 

SE  Model  for  SoS  Based  on 
7  Core  Elements  of  SoS  SE 


Translating 

capability 

objectives 


I 


Understanding 
systems  & 
relationships 


Orchestrating 
upgrades 
to  SoS 


Addressing 
requirements 
&  solution 
options 


Monitoring 
&  assessing 
changes 


Assessing 
performance 
to  capability 
objectives 


Developing 
&  evolving 
SoS 

architecture 


New 
SoS  SE 
role 


Persistent 
SoS  overlay 
framework 


SoS 

upgrade 

process 


External 

influences 


External  Environment 


SE  Processes  Support  Core  Elements 


DoD  Defense  Acquisition  Guide 
presents  1 6  basic  SE  processes 

In  an  SoS,  SE  team  adapts 
these  processes  to  execute 
core  SE  elements 

Focus  for  SoS  SE  is  on 
technical  management  since 
implementation  is  in  systems 


SoS  SE 
Core 
Elements 


TECHNICAL  PROCESSES 
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&  relationships 
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Developing  &  evolving 
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and  solution  options 
Orchestrating  upgrades 
to  SoS 
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Core  Elements  of  SoS  SE  <i  of  3) 


Translating 

capability 

objectives 


•  Translating  SoS  capability  objectives  into  high 
level  requirements  over  time 

•  SoS  objectives  based  on  broad  capability  objectives 

•  SE  team  plays  strong  role  in  establishing  requirements 
and  understanding  dynamics  of  the  environment 


Understanding 
systems  & 
relationships 


•  Identifying  and  understanding  the  systems 
that  impact  SoS  objectives 

•  Focus  on  components  and  dynamics  vs  boundaries 

•  Extends  beyond  technical  to  broader  context  of 
management,  organizational,  development  plans, 
funding,  etc. 


Monitoring 
&  assessing 
changes 


•  Anticipating  and  assessing  impacts  of 
potential  changes  on  SoS  performance 

•  Given  scope  of  SoS  authority,  key  to  SoS  SE  is 
identifying  and  addressing  changes  in  systems  and  other 
areas  (e.g.  threat)  which  may  impact  the  SoS 


Core  Elements  of  SoS  SE  (2  of  3) 


Developing 
&  evolving 
SoS 

architecture 


Developing  and  evolving  SoS  architecture 


•  This  includes 

•  Concept  of  operations 

•  Systems,  functions  and  relationships  and  dependencies, 
both  internal  and  external 


•  End-to-end  functionality,  data  flow  and  communications 
within  the  SoS. 


•  Provides  the  technical  framework  for  assessing  options 
and  implications  for  meeting  requirements  over  time 

•  Persistence,  tolerance  for  change 


•  An  architecture  is  the  structure  of  components,  their  relationships, 
and  the  principles  and  guidelines  governing  their  design  evolution 
over  time  (IEEE  Std  610.12  and  DoDAF). 

•  The  architecture  of  an  SoS  is  a  persistent  technical  framework  for 
governing  the  evolution  of  an  SoS  over  time. 


Core  Elements  of  SoS  SE  <3  of  3) 

•  SoS  requirements  and  solution  options 

•  Requirements  addressed  at  both  SoS  &  systems 

•  Recommend  SoS  requirements  based  on  both  priority  and 
practicality 

•  SoS  and  system  SE  teams  identify  and  assess  options 

•  Result  is  plan  for  development  for  next  increment 

•  Orchestrating  SoS  Upgrades 

•  Upgrades  implemented  by  systems  under  system  SE  teams 

•  SoS  SE  team  plans,  facilitates,  integrates  and  tests 
upgrades  to  the  SoS 

•  Development  based  on  incremental  approaches  (bus  stop, 
wave)  which  accommodate  asynchronous  system 
developments 

•  Assessing  SoS  Performance 

•  Based  on  measures  of  SoS  user  results  applied  in  different 
settings  (test,  exercises,  M&S,  operations) 

•  Opportunity  to  identify  changes  and  emergent  behavior 


Assessing 
performance 
to  capability 
objectives 


Orchestrating 
upgrades 
to  SoS 


Addressing 
requirements 
&  solution 
options 


View  of  SoS  Upgrade  a  of  2) 


For  each  increment 


Recommend 
rqts  for  this 
increment 


L 


- .  Addressing 

1  “candidate  requirements 
syuspportto  &  solution 

functions  options 
Assess  options 


jpport 
functions 


Develop  plan 


Assessing 
performance 
to  capability 
objectives 


Orchestrating 
upgrades 
to  SoS 


Assess  SoS 


Validate  sets 
of  systems 


SoS 


Coordinate,  monitor  and  facilitate  systems’ 
development,  test  and  evaluation 


View  of  SoS  Upgrade  (2  of  2) 


Translating 

Capability 

Objectives 


Monitoring  & 

Assessing 

Changes 

L  A 

Understanding 
Systems  8t 
relationships 


Developing 
&  Evolving 
SoS 

Architecture 


Assessing 

SoS 

Performance 


Addressing 
requirements 
&  solution 
options 


Orchestrating 
upgrades 
to  SoS 


SoS 


Systems 
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Multiple,  possibly  concurrent  increments 


Guide  Extract 

Relationships  Among  the  Core  Elements 


Understanding 
systems  & 
relationships 


t 


Input: 

Changes  which  impact 
systems  and  relationships 


systems 

Output: 


Status  of  systems, 
relationships,  and 
functionality 


l 


Output: 

Status  of  systems, 
relationships,  and 
functionality 


Input: 

First  order  SoS 
objectives  and 
expectations 

Output: 

Status  of  systems, 
relationships,  and 
functionality 

Input: 

Updated  architecture 
information 

Output:  — 

Status  of  systems, 
relationships,  and 
functionality 


Input: 

Upgrades  which 
impact  systems  and 
lips 


Translating 

capability 

objectives 


Developing 
&  evolving 
SoS  architecture 


\ 


Orchestrating 
upgrades  ' 
to  SoS 


Monitoring 
&  assessing 
changes 


Addressing 
requirements 
&  options 


Guide  Extract:  SE  Processes 
Supporting  Each  SoS  SE  Element 


Technical  or  Technical 

Relationship  to  SoS  SE  Core  Element 

Management  Process 

Logical  Analysis  is  the  process  of 
obtaining  sets  of  logical  solutions  to 
improve  understanding  of  the  defined 
requirements  and  the  relationships  among 
the  requirements  (e.g.,  functional, 
behavioral,  temporal). 

Logical  Analysis  is  a  key  part  of  Understanding  Systems  and 
Relationships.  Basic  to  engineering  an  SoS  is  understanding  how  systems 
support  SoS  functionality.  In  developing  a  new  system,  the  systems  engineer 
allocates  functionality  to  system  components  based  on  a  set  of  technical 
considerations.  In  an  SoS,  the  systems  engineer  develops  an  understanding  of 
the  functionality  extant  in  the  systems  and  how  that  functionality  supports  SoS 
objectives,  as  a  starting  point  for  SoS  architecture  and  evolution.  ... 

Risk  Management ...  helps  ensure 
program  cost,  schedule,  and  performance 
objectives  are  achieved  at  every  stage  in 
the  life  cycle  and  to  communicate  to  all 
stakeholders  the  process  for  uncovering, 
determining  the  scope  of,  and  managing 
program  uncertainties. 

Risk  management  is  a  core  function  of  SE  at  all  levels.  In  Understanding 
Systems  and  Relationships,  the  systems  engineer  assesses  the  current 
distribution  of  functionality  across  the  systems  and  identifies  risks  associated 
with  either  retaining  the  status  quo  or  identifying  areas  where  changes  may 
need  to  be  considered.  The  systems  engineer  also  considers  approaches  to 
monitor,  mitigate,  or  address  risks.  Such  risks  might  include  ... 

Configuration  Management  is  the 

application  of  sound  business  practices  to 
establish  and  maintain  consistency  of  a 
product's  attributes  with  its  requirements 
and  product  configuration  information. 

Understanding  Systems  and  Relationships  is  where  the  CM  process  for 
the  “as  is”  SoS  resides  and  is  maintained  as  the  SoS  product  baseline.  In  a 
system  the  CM  process  addresses  all  of  the  ‘product’s’  features  where  the 
system  itself  is  the  product.  In  an  SoS,  the  ensemble  of  systems  and  their 
functionality  is  the  product;  the  SoS  CM  depends  on  the  CM  of  the  systems  to 
maintain  much  of  the  product  information,  since  the  system  owner,  PM,  and 
system  systems  engineer  normally  retain  responsibility  for  their  systems.  The 
SoS  CM  focuses  on  the  linkage  to  the  system  CM  and  crosscutting  attributes 
which  pertain  to  the  SoS  not  addressed  by  the  CM  of  the  systems.... 

What  is  Working? 
SoS  SE  Principles 


•  Address  organizational  as  well  as  technical  perspectives 

-  Factor  in  broader  set  of  consideration  into  trade  space  and  technical 
planning 

•  Focus  on  areas  critical  to  the  SoS 

-  Leave  the  rest  to  the  systems  engineers  of  the  systems 

•  Technical  management  approach  reflects  need  for 
transparency  and  trust  with  focused  active  participation 

•  SoS  designs  are  best  when  open  and  loosely  coupled 

-  Impinge  on  the  existing  systems  as  little  as  possible 

-  Are  extensible,  flexible,  and  persistent  overtime 

•  Continuous  (‘up  front’)  analysis  which  anticipates  change 

-  Design  strategy  and  trades  performed  upfront  and  throughout 

-  Based  on  robust  understanding  of  internal  and  external  sources  of 
change 


Way  Ahead 


Guide  is  out  and  in  use,  offers  a  first  step 

-  Highlights  the  issues  of  SoS  in  DoD  today 

-  Provides  some  support  for  SE  teams  operating  in  SoS  today 

-  Plan  for  outreach  and  educational  materials 

-  Assess  added  guidance  for  areas  such  as  Systems  Engineering  Plans 
Efforts  are  underway  to  support  update  to  the  guide 

-  A  follow-up  data  collection  to  get  an  understanding  of  ‘how  to’  level 
of  information  from  ongoing  SoS  SE  efforts 

-  Cooperative  effort  with  NDIA  M&S  Committee  to  examine  promise  and 
experience  with  M&S  to  support  SoS  SE 

-  Series  of  industry  exchanges  on  SoS  topics  of  common  interest 

-  International  cooperative  efforts  are  being  initiated 

-  Expansion  into  broader  areas 

•  SE  for  Capability  Portfolio  Management 

•  Net  Centric  Enterprise  Systems/Services 


Backup 


Active  SoS  SE  Practitioners 


Name 

Acronym 

Owner 

Approach 

Army  Battle  Command  System 

ABCS 

Army 

Acquisition  Program 

Air  Operations  Center 

AOC 

Air  Force 

Acquisition  Program 

Ballistic  Missile  Defense  System 

BMDS 

Joint 

Acquisition  Program 

USCG  Command  &  Control  Convergence 

C2  Convergence 

Coast  Guard 

Strategy 

Common  Aviation  Command  &  Control  System 

CAC2S 

Marine  Corps 

Acquisition  Program 

Distributed  Common  Ground  Station 

DCGS-AF 

Air  Force 

Program  Office 

DoD  Intelligence  Information  System 

DoDIIS 

Intel 

DIA  CIO  Initiative 

Future  Combat  Systems 

FCS 

Army 

Program  Office 

Ground  Combat  Systems 

GCS 

Army 

Program  Executive  Office  PEO 

Military  Satellite  Communications 

MILSATCOM 

Joint 

AF  Wing 

Naval  Integrated  Fire  Control  -  Counter  Air 

NIFC-CA 

Navy 

SE  Integrator  in  PEO 

National  Security  Agency 

NSA 

Intel 

Agency 

Naval  Surface  Warfare  Center  Dahlgren 

NSWC 

Navy 

Warfare  Center 

Single  Integrated  Air  Picture 

SIAP 

Joint 

Acquisition  Program 

Space  and  Missile  Systems  Center 

SMC 

Air  Force 

SE  Authority 

Space  Radar 

SR 

Joint 

Acquisition  Program 

Theater  Joint  Tactical  Networks 

TJTN 

Joint 

PEO 

Theater  Medical  Information  Systems -Joint 

TMIP 

Joint 

Acquisition  Program 

Provided  a  basis  for  understanding  SoS  in  DoD  Today 
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Sustainment  Environment 


727th  Aircraft 
Sustainment 
Wing 


PROVIDING 


-  071  Vi 


EFFICIENT  WEAPON 


; 1LJPPOPT 


Col.  Paul  Waugh 

Commander 


n% 


Mr.  Bob  Valdez 

Deputy  Director 


Mr.  James  Miller 

Director  of  Engineering 


OC-ALC  Wing  Structure 


OC-ALC  COMMAND  SECTI  ON 


OC-ALC  STAFF  OFFI CES 


327th  Aircraft 
Sustainment  Wing 


327th  Aircraft 
Sustainment 
Group 

727th  Aircraft 
Sustainment 

1  Group 

427th  Aircraft 
Sustainment 
Group 

747th  Aircraft 
Sustainment 

1  Group 

639th  Aircraft 
Sustainment 
Group 

827th  Aircraft 
Sustainment 

1  Group 

559th  Aircraft 
Sustainment 
Squadron 


76th  Maintenance 
Wing 

76th  Aircraft 
Maintenance 
Group 

76th  Propulsion 
Maintenance 
Group 

76th  Commodities 
Maintenance 
Group 

76th  Software 
Maintenance 
Group 

76th  Maintenance 
Support 
Group 


72d  Air  Base  Wing 

72d  Mission 
Support 
Group 

72d  Medical 
Group 


327th  Aircraft  Sustainment  Wing 


327th  ASW  Responsibilities 


r  1503 
Aircraft  Mgd 
(357  Inactive) 


M  1382 
Air  Traffic 
Control  & 
Landing  Sys 
Mgd 


28,000+ 
Engines  Mgd 
51  types 


62  Weapon 
Systems 
33 

ATCALS 


24 

Commands 


327 

ASW 


212  USAF 
Bases 
41  FMS 
Nations 


FY07 

153  ProgranN  f 

Depot 

Maintenance  FY( 

Completed  $3. 


$14. 8B 
Contracts 
Managed 
I  n  FY07 


So  What  is  the  Airworthiness  Problem? 


•  Airworthiness  is  a  requirement  for  all  aircraft,  whether  FAA  or  DoD 

•  Tinker  AFB  manages  20-plus  different  types  of  CDA 

-  Aircraft  use  a  mixture  of  FAA  and  Air  Force  criteria  and  methods  of  compliance  to 
verify  airworthiness  when  modifying  the  aircraft 

•  Modifying  a  CDA  by  a  process  that  combines  both  FAA  Certification 
and  Air  Force  Certification  could  result  in  a  hybrid  safety  standard. 

-  Such  a  standard  is  unproven  by  either  the  FAA  or  the  DoD,  and 
could  therefore  put  the  aircraft  and  crew  at  risk 

•  No  planning  and  implementation  process  to  ensure  comprehensive 
and  complete  airworthiness  of  all  designs  and  parts 

•  No  tracking  the  organization’s  progress  regarding  airworthiness  for 
upper  management  in  a  fleet  of  over  400  aircraft  throughout  the 
entire  lifecycle  of  the  CDA 


Airworthiness  Project  Overview 


*  Problem  Statement 

-  Current  practices  do  not  ensure  100%  of  CDA 
modification  design/parts  are  correctly  certified  for 
airworthiness. 

*  Project  Definition  and  Scope 

-  727  ACSG  aircraft  (CDA)  sustained  by  Boeing 

-  Airworthiness  certification  to  cover  various  (FAA  & 
Military)  compliance  methods 

-  Review  and  “Walk”  the  entire  process  in  both  orgs 

-  Define  Responsibility  Accountability  Authority  (RAA) 
for  any  process  decision  pts 

-  Ensure  certification  means  supports  lifecycle 
sustainment 

-  Must  include  metrics  for  upper  management  visibility 


GAPS 


•  Government  does  not  clearly  state 
airworthiness  requirement  to  contractors 

•  Responsibility,  Accountability  and 
Authority  (RAA)  not  well  defined  by  FAA, 
Government  or  Contractor 

•  No  comprehensive  airworthiness 
certification  plan 

-  Plan  not  done  early  in  modification  process 

-  Plan  not  coordinated  between  Government, 
FAA  and  Contractor 

•  No  control  mechanisms  in  place  to 
measure  airworthiness 


Gap  #1:  Requirements  Not  Clear 


•  Airworthiness  very  briefly  mentioned 

•  Rarely  states  what  type  airworthiness 
certification  required 

•  Rarely  addessses  parts 

•  Rarely  addresses  life  cycle 
cost/sustainment  aspects 

•  Does  not  address  who/when  airworthiness 
decisions  will  be  made 

•  Examples.... 


Airworthiness  SOW  Language  Examples 


•  “The  contractor  shall-obtain  FAA  approval  for  this 
modification...” 

•  “Any  equipment  installed  as  part  of  this  modification 
not  covered  with  full  FAA  certification  must  be...” 

•  “Obtain  FAA  approval  for  engineering  drawings...” 

•  “This  SOW  directs  the  contractor  to  provide  an  FAA 
approved  modification... 

•  “Contractor  shall  obtain  FAA  approval  where 
applicable... 

•  “Contractor  shall  obtain  FAA  where  practical...” 


Gap  #2:  RAA  Not  Well  Defined 


•  Responsibility,  Accountability  and 
Authority  (RAA)  not  well  defined  by  FAA, 
Government  or  Contractor 

•  Neither  Gov’t  nor  Contractor  have  policy 
in  place  defining  who  makes 
airworthiness  decisions  throughout 
process 

-  Design:  Not  clear  who  decides  which  of  design 
cert  will  be  followed 

-  Parts:  Decisions  made  at  various  levels,  part 
“pedigree”  often  assumed,  or  not  given 
consideration  to  life  cycle  cost 


GAP  #3:  No  Certification  Plan 


•  MIL-HDBK-516B  describes  criteria,  but  not 
implementation  and  planning 

•  Currently  no  certification  plan  required  for 
modification 

•  No  plan  provided  up-front  regarding  all 
designs  and  all  parts 

•  Government  usually  does  not  find  out 
until  end  what  the  certification  is 


GAP  #4:  No  Control  Measures 


•  How  much  FAA  certified  and  how  much 
Military  certified? 

•  Which  design  certification  methods  used? 

•  What  are  the  pedigrees  of  all  the  parts? 

•  Does  the  actual  delivered  modification 
match  the  planned? 

•  How  can  you  keep  your  SPM  and  Chief 
Engineer  informed  of  this  important  topic 
before  the  signing  of  the  DD  Form  250? 


So  What  Are  Doing  About  It? 


Instigated  a  step-by-step  Operating  Instruction 
to  implement  air  worthiness  management 
throughout  the  organization 

Implemented  tangible  approach  that  is: 


-  Aimed  at  the  working  level 

-  Applies  to  both  contractor  and  Air  Force 

-  Applicable  throughout  entire  organization 

-  Accounts  for  status/progress  through  metrics 

-  Always  starts  with  requirements 


4  Solution  Recommendations 


•  Improve  SOW  wording  (Requirements) 

•  Complete  airworthiness 
approach/certification  plan  for  both  design 
and  parts  early 

•  Clearly  define  decision  making  authority  for 
each  airworthiness  condition 

•  Establish  control  measures  to  verify  100% 
certification  of  designs  and  parts  and  keep 
upper  management  informed 


1 


Sol’n  #1:  Improved  SOW  Words 


•  Ol  contains  decision  tree  which  will  drive 
appropriate  level  of  airworthiness  requirements 

•  Airworthiness  certification  requirements 
expanded  and  clarified  to  contractor 

•  Ol  contains  “cut-and-paste”  template  SOW 
language  for  modification  contracts 

•  Templates  available  for: 

-  FAA  Airworthiness  Certification 

-  Non-FAA  Airworthiness  Certification 

-  Airworthiness  Sustainment  Requirements  (Parts) 

-  Airworthiness  Documentation 
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Sol’ri  #2:  Airworthiness  Cert.  Plan 


•  The  Airworthiness  Certification  Plan  Must: 

-  Be  delivered  NLT  System  Requirements  Review 

-  Cover  100%  of  planned  design 

-  Cover  100%  of  planned  parts 

•  Instructions  for  Continued  Airworthiness  (ICA) 

•  Sustainment  plan  to  ensure  availability  of  airworthy  parts 
throughout  life  cycle 

-  For  all  non-FAA  parts  or  design,  must  have  SPM  or 
Chief  Engineer  approval 

-  Account  for  life  cycle  maintenance 

-  Deliver  applicable  airworthiness  certification 
documentation 

-  Include  specific  control  measures  (metrics)  to  track 
health 


Sol’n  #3:  Decisions  at  Right  Level 


*  Clearly  define  decision  making  authority 
for  each  airworthiness  condition 

•  01  contains  detailed  matrix  for  each  certification 
method,  part  certification  and  documentation 
requirement 

•  01  clearly  defines  for  each  condition  what  level  has 
approval  authority 

-  Chief  Engineer  or  Single  Manager 

-  Engineering  Flight  Director 

-  Lead  engineer  or  program  manager 

•  Boeing  make  similar  changes  to  their  internal 
processes 


Sol’n  Gap  #4:  Developed  Metrics 


•  Establish  control  measures  to  track  the 
following: 

-  Design/part  certification  method 

-  Design  certification  breakout 

-  Part  certification  breakout 

*  Start  tracking  at  beginning  and  continue 
through  delivery 

•  Brief  to  Upper  Management  Quarterly 

•  Metrics  must  have  ability  to  roll-up 

•  For  a  collection  of  modifications 

•  For  entire  aircraft 

•  For  entire  organization 


Design/Part  Certification  Method 


PARTS 


DESI GN 


Design  Certification 


Total  Mods 


HilCertlnriteddov 


Breakout 


.1 


Part  Certification  Breakout 


New  Process  to  Ensure  Airworthiness 


rService  literature^ 
Obsolescence/ 
Parts  Sub 
Request 


USAF/  SPO 
(Customer) 


^Mil-HDBK-516^ 
AFPD  62-6 
TACC 

1067,  ORD 


Notify 

Contractor 


Approve 

8130-31 


r  Control  Mechanisms 


SOW 

EST 


8130-31 

Acceptance 


Contractor 


SOW/Contract 


Develop 
Certification 
Approach 

[Control  Mechanisms! 


Define 

Affected 

T.O.'s 


Generate 
Design  Data 


Assy 
Completion, 
I  nspection 
&  Test 


A/C  Instl, 

I  nspection 
St  Test 


A/C 

Testing, 

|  T.O.  Validation,  | 
I  nspection 


Final  Data 
Submittals 


Field  or  Depot 
Maintenance 


T.O. 

Updates 


Cert  Plan 
8130-31 


Type  Design 
MDL 
Reports 


ICA 

(T.O.'s) 


FAA 
and/  or 


Type  Des 
Report: 
MDL 


STC 

8130-31 


Strengthened  SOW  language,  defined  intent  and 

established  clear  RAA 


Ensured  cert  approach  in  place  before  SRR 


I  mplemented  control  measures  (metrics)  to 
verify  both  designs  and  parts 


Approve 
Type  Design 
Change 


Update 

STC 


Return  to 

Service 

Return  to 

Service 

8110-12 


8100-9 

8120-10 

8130-3 

8100-1 


STC 
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LEGEND 


© 


©  Record  A 


Control 

Point 


Summary 


*  Focuses  on  airworthiness  certification  planning 
and  implementation  rather  than  establishment  of 
airworthiness  certification  criteria 

*  Provides  a  standardized  proactive  airworthiness 
certification  management  process  consistent 
with  Air  Force  policy 

*  Provides  a  process  to  ensure  airworthiness 
certification  requirements  are  an  integral  part  of 
program  management — contractor  and  DoD 

*  Ensures  “the  right”  airworthiness  certification 
requirements,  for  both  design  and  parts,  are 
identified,  implemented,  monitored,  controlled, 
and  reported. 


Wi 


Questions  ? 


Current  Process 


Field  01 

Maintc 

— H 

r  Depot 
E?nance  1 

1 

r 

T.O. 

Updates 

f 

Approve 

Type  Design 
Change 

_ 5 

i _ 

Update 

STC 

Parking  Lot  Gaps 


(G1)  MACC’s  not  being  prepared  for  each  modification 

(Gla)  Cert  plans  that  are  generated  by  contractor  are  not  coordinated  with  Government 


RCM  Templa 


Event 

0  Effort  kickoff  or  major  review/change 


1  Overall  Certification 


2  Are  there  portions  of  the  modification  which 

can/should  be  fully  FAA  certified?  That  is 
elements  (A)  which  are: 

•  Similar/identical  to  widespread  commercial 
requirements 

•  Similar  to  private  initiatives  in  effects  on 
airworthiness,  flight  characteristics, 
operational  characteristics,  or  pilot  technique 

•  Are  similar  to  private  initiatives  in  aircraft 
usage  or  implementation  of  mission  or 
interior  accommodations 

•  Can  meet  all  applicable  FAA  regulations  and 
the  same  requirements  for  a  commercial 
modification 

3  Are  there  adaptations  or  alterations  of  commercial 

aviation  equipment  required  to  suit  military 
or  mission  requirements? 

3  Will  existing  STCs  (S)  be  partially  changed  as  a 
a  result  of  this  modification? 


Step  1 


Step  1 


Identify  scope  of  modification,  including 


functions/  capabilities  affected/incorporated, 
major  hardware  elements  and  LRUs,  areas  of  a/c 
affected,  and  system  or  systems  involved. 

R1  -  Prepare  an  integrated  airworthiness  Step  2  Step  2 

certification  plan  to  accomplish  comprehensive 
design  certification. 

R2  -  Provide  Instructions  for  Continued 
Airworthiness  to  permit  aircraft  sustainment  in 
accordance  with  certified  design 
R22  -  Provide  control  measures  (metrics)  to 
track  design/part  certification  method,  part 
certification  breakout  and  design  certification 
breakout  on  or  before  SRR  with  updates  to 
metrics  throughout  modification  program 
R23  -  Provide  delivery  dates  for  metrics  and 
supporting  data  in  program  integrated  master 
schedule 

R3  -  Obtain  FAA  approval/certification  for  (A)  Step  3  Step  5 

equipment/  capability  implementation  in 

accordance  with  requirements  applicable  to 

aircraft  operating  under  FAR  Part  (91,  121,  etc. 

as  applicable). 


•  R4  -  Modify  (E)  to  provide  capabilities  (Z)  Step  3a  Step 

•  R5  -  Obtain  FAA  certification  for  (E),  as  modified  3 

a 

•  R18  -  Obtain  FAA  approval  of  changes  to  (S) 

Gov’t  note:  Military  a/c  primarily  don’t  maintain  the 

airworthiness  certificate  (from  the  strict  FAA 

DorAmmonrl  thnt  o  toAhniml  riel/ 


RCM  Template 


Event  Requirement 


5  Are  there  elements  of  the  modification  which 
cannot  be  approved  for  carriage  by  the  FAA  (B)? 
Examples  include: 

•Hazardous  materials  or  equipment 
•Equipment  which  cannot  be  demonstrated  to  be 
safe  even  when  not  operating 

•R6  -  Obtain  Provisions  Only  FAA 
approval/certification  of  interfaces/provisions  for  (B). 

Step  6 

Step  6 

6  Will  military  qualified  equipment  (C)  be 
needed/used  in  the  modification? 

•R7  -  Obtain  FAA  installation  certification/approval  for 
(C)  using  military  qualification  and  operational  data. 

•R8  -  Perform  necessary  analysis  to  support  FAA 
certification/approval  for  (C) 

•R9  -  Perform  additional  testing  required  to  support 

FAA  certification/approval  for  (C) 

Step  7 

Step  7 

7  Will  the  modification  use/apply  non-aviation 

commercial-  or  consumer-grade  equipment 

•RIO  -  Perform  safety  analyses  covering  use  and 
operation  of  (L) 

•R11  -  Obtain  FAA  certification/approval  for  (J) 

•R  12  -  Identify  any  equipment  in  (L)  which  is  unsafe  or 
hazardous  when  applied  to  this  modification  (H) 

Step  8 

Step  8 

8  Is  there  hazardous  commercial/consumer 

equipment? 

•R13  -  Design  enclosures  and/or  accommodations  to 
control  hazards  posed  by  (H) 

•R14  -  Obtain  FAA  certification/approval  for 
enclosures  and/or  accommodations  for  (H) 

Step  9 

Step  9 

9  Is  there  doubt  that  sustainment  parts  and  repairs 

can  be  readily  obtained  for  FAA  certified  design, 
throughout  the  life  of  the  modification? 

•R15  -  Develop  a  sustainment  plan  to  ensure 
availability  of  FAA  parts  repair  capability  throughout 
the  life  of  the  modification 
•R16  -  Develop  a  sustainment  plan  to  ensure 
availability  of  FAA  replacement  parts  throughout  the 
life  of  the  modification 

Gov’t  note:  Requires  a  Logistics  Support  Analysis  to 
determine  right  path  FAA  or  not  -  don’t  assume  pure 

FAA  is  the  right  approach. 

Step  10 

Step 

10 

RCM  Template 


Event  Requirement 


10  Are  there  elements  (M)  that  will  not  be  FAA 

certified? 

•R17  -  Develop  a  comprehensive  plan  to  certify  (M)  in 
accordance  with  military  airworthiness  certification 
requirements  (MIL-HDBK-516) 

Step  1 1 

Stop 

1 1  Are  there  elements  B? 

•R18  -  Conduct  analyses,  tests,  and  demonstrations  to 
qualify  (B) 

•R19  -  Prepare  and  submit  data  to  support  certification 
of  (B)  for  airworthiness,  including  operation  in-flight 

Step  12 

Step  12 

12  Are  there  elements  K? 

•R20  -  Conduct  analyses,  tests,  and  demonstrations  to 
demonstrate/develop  safe  installation  and  use  of  (K) 

•R21  -  Prepare  and  submit  data  to  support  certification 
or  approval  of  (K)  for  installation  and  use 

Step  13 

Step  13 

13  Military  Certification 

•R21  -  Conduct  necessary  analyses,  test,  and 
demonstrations  to  support  airworthiness  and 
operations  approval  for  (M) 

RCMTemplateKey 


•  A  Elements  of  modification  which  may  receive  full  FAA 
certification/approval 

•  B  Military  only  elements  of  the  modification  -  those  which  cannot  be 
approved  for  installation  by  FAA  and  require  provisions  only  approval 

•  C  Military  qualified  equipment  for  which  FAA  certification  may  be 
obtained 

•  E  Commercial  aviation  equipment  which  must  be  altered  or  adapted  to 
meet  military  requirements  (subset  of  A) 

•  H  Non  aviation  commercial  or  consumer  equipment  which  is  unsafe  or 
poses  hazards  which  cannot  be  mitigated  (subset  of  L) 

•  J  Non  aviation  commercial  or  consumer  equipment  which  may  be  FAA 
certified  (subset  of  L) 

•  K  Non  aviation  commercial  or  consumer  equipment  which  cannot  be 
FAA  certified  or  for  which  accommodations  cannot  be  designed  to  permit 
certification  (subset  of  L  and  possibly  H) 

•  L  Non  aviation  commercial  or  consumer  equipment  needed/used  as 
part  of  modification 

•  M  Elements  requiring  military  airworthiness  certification  (Includes  B 
and  K) 

•  S  Existing  STCs  modified  in  the  course  of  the  current  modification 

•  Z  Capabilities  or  features  for  military  purposes  which  must  be 

incorporated  into  commercial  aviation  equipment 


Basic  Systems  Engineering  Process 


INPUTS 


Requirements 

Analysis 


Requirements 

Loop 


Verification 

Loop 


Functional 

Analysis/ 

Allocation 


Design 

Loop 


Analysis  & 
Control 


Design 

Synthesis 


OUTPUTS 


Major  Modification  Programs 


17  Current  Programs  m 

Q  KC-10  AMP  -  ASC  Lead  (ACAT  II)  $1.03B 

g  KC-10  Dual  406  MHz  ELT  Upgrade  (ACAT  III)*  $2.4M 

g  KC-10  Iridium  Phone  (ACAT  III)*  $2.7M 

g  KC-10  UHFSATCOM  Antenna  (ACAT  III)*  $2.6M 

g  VC-25  Forward  Lower  Lobe  (FLL)  Cooling  (ACAT  III)  $14.4M 

g  VC-25  Presidential  Data  System  (PDS)  (ACAT  III)*  $223.3M 

g  VC-25  CNS/ATM  (ACAT  III)*  $41 .8M 

g  C-20  Gulfstream  Test  Vehicle  (GTV)  (ACAT  III)*  $8.7M 

g  E-9  Telemetry  Sys  Upgrade  (ACAT  III)*  $5.9M 

g  E-4B  Mod  Block  I  (ACAT  II)  *  $421. 4M 

g  E-4B  256  Kbps  High  Speed  Data  via  INMARSAT  (ACAT  III)*  $8.4M 

R  C-12EFIS  (ACAT  III)  $77.7M 

D  HFGCS  Network  Control  Station  -  West  (ACAT  III)*  $23.2M 

D  HFGCS  AFSPC  Test  Range  HF  Modernization  (ACAT  III)*  $3.9M 

G  HFGCS  Network  Optimization  -  Spiral  II  (ACAT  III)*  $7.1  M 

6  HFGCS  Navy  Consolidation  (ACAT  III)*  $6.4M 

G  HFGCS  Audit  Log  Upgrade  (ACAT  III)*  $189K 


Systems  Engineering 
Performance  Measures 

22  Oct  08 


Jim  Miller 

Director  of  Engineering 
727  ASW/EN 
Phone:  (405)  736-4101 
james.c.miller@tinker.af.mil 


Sustainment  Environment 


W  i  i 


727th  Aircraft 
Sustainment 
Wing 

Col.  Paul  Waugh 

Commander 

Mr.  Bob  Valdez 

Deputy  Director 


Mr.  James  Miller 

Director  of  Engineering 


PROVIDING  EFFECTIVE  &  EFFICIENT  WEAPON  SYSTEM 


EC 


OC-ALC  Wing  Structure 


OC-ALC  COMMAND  SECTION 


OC-ALC  STAFF  OFFICES 


327th  Aircraft 
Sustainment 

Group  1 

727th  Aircraft 
Sustainment 

I  Group 

427th  Aircraft 
Sustainment 

Group  1 

747th  Aircraft 
Sustainment 

1  Group 

639th  Aircraft 
Sustainment 

Group  I 

827th  Aircraft 
Sustainment 
|  Group 

559th  Aircraft 
Sustainment 
Squadron  1 

Maintenance 

Group 


76th  Propulsion 
Maintenance 
Group 

76th  Commodities 
Maintenance 
_ Group _ 

76th  Software 
Maintenance 
Group 

76th  Maintenance 
Support 
Group 


Group 

72d  Medical 
Group 


327th  Aircraft  Sustainment  Wing 


327th  Aircraft  Sustainment  Wing 
Col  Paul  Waugh,  736-5865 
Bob  Valdez,  736-5865 
Jim  Miller,  736-4101 


Wing  Group  Sqdn  Div 


747  ACSG 
Combat  Sys 

Col  Keith  Weyenberg  739-3448 


556  A/C  Sust  Sq 
(B-2  Bomber) 

Col  Mark  Hays  739-4260 
Vacant  739-4260 


827  ACSG 
C/KC-135 

Col  James  Nally  736-7755 
Ralph  Garcia  736-7755 

550  A/C  Sust  Sq 
(Combat  Support) 
Edward  Durell  736-5391 


551  A/C  Sust  Sq 
(Modification) 

Sherie  Donahay  736-5188 
Maj  Jason  Englund  739-8357 


327th  ASW  Responsibilities 


;4?/i\\\ys 


^  1503 

Aircraft  Mgd 
(357  Inactive) 


w  1382 
Air  Traffic 
Control  &  ^ 

Landing  Sys  } 
Mgd 


28,000+ 
Engines  Mgd 
51  types 


62  Weapon 
Systems 
33 

ATCALS 


24 

Commands 


212  USAF 
Bases 
41  FMS 
Nations 


FY07^ 
153  Program 
Depot 

Maintenance 

Completed 


FY07 

$3.3B 

Obligation 

Authority 


$14. 8B 
Contracts 
Managed 
I  n  FY07. 


So  What  is  System  Engineering? 


Everything  Can  Be  System  Engineering 


AF  and  DoD  Sys  Eng  Policy 
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SE  Policy  Addendum 

Signed  by  the  Marvin  R-  Sambour,  Asst.  SecAF  (Acquisition)  Apr  03  &  Jan  04 


•  Policy  Memo  03A-005,  9  Apr  03 

-  Subj:  Incentivizing  contractors  for  Better  Systems 
Engineering 

-  “An  immediate  transformation  imperative  for  all  our 
programs  is  to  focus  more  attention  on  the  application 
of  Systems  Engineering  principles...” 

-  Directing  the  following: 

•  A.  Assess  ability  to  incentivize  contractors  to  perform 
robust  SE 

•  B.  Develop  SE  performance  incentives 

•  C.  Include  SE  processes/practices  during  all  program 
reviews 

•  Policy  Memo  04A-001,  7  Jan  04 

-  Subj:  Revitalizing  Air  Force  and  Industry  Systems 
Engineering  (SE)  -  Increment  2 

-  “...intended  to  institionalize  key  attributes  of  an 
acceptable  SE  approach  and  outcome...” 

-  “...must  focus  on  an  end  state...” 


Systems  Engineering  Policy  in  DoD 


•  All  programs,  regardless  of  ACAT  shall: 

-  Apply  an  SE  approach 

-  Develop  a  Systems  Engineering  Plan  (SEP) 

•  Describe  technical  approach,  including  processes, 
resources,  and  metrics 

•  Detail  timing  and  conduct  of  SE  technical  reviews 

*  Director,  DS  tasked  to  provide  SEP  guidance  for 
DoDI  5000.2 

-  Recommend  changes  in  Defense  SE 

-  Establish  a  senior-level  SE  forum 

-  Assess  SEP  and  program  readiness  to  proceed  before 
each  DAB  and  other  USD(AT&L)-led  acquisition  reviews 


So  What  is  the  Problem? 


•  High-level  policy  is  there,  But ... 

-  How  do  you  know  if  you  are  doing  it? 

-  How  do  you  measure  so  you  drive  the  behavior? 

•  Sys  Eng  scope  can  be  huge,  So  ... 

-  What  tenets  should  be  measured? 

-  What  are  the  key  characteristics? 

-  How  can  it  apply  across  different  programs  and 
organizations? 

•  Sys  Eng  is  important,  Yet ... 

-  No  accepted,  standard  metrics 

-  No  measure  of  sys  eng  current  status 

-  No  metrics  for  both  PM  and  upper  management 


Why  Measure  Systems  Engineering? 


•  When  performance  is  measured  ... 
performance  improves 

•  When  performance  is  measured  and 
reported  ...  the  rate  of  performance  improves 

•  When  performance  is  measured,  reported, 
and  compared  ...  the  rate  of  performance 
continues  to  improve 


Sys  Eng  Metrics  Key  Characteristics 


•  Must  Measure  Major  Components  of  Sys  Eng 

•  Must  Be  Few  in  Number 

•  Must  Avoid  Extensive  Data  Collection  Efforts 

•  Must  Describe  Current  Status,  Not  Lagging 

•  Must  Be  Targeted  for  Management 

•  Must  Allow  For  Comparison  Between  Programs, 
Organizations,  and  Time 

•  Must  Be  Cumulative  (Ability  to  Roll-Up) 


What  Was  Our  Approach? 


•  Defined  first  5  Sys  Eng  Tenets 

•  Step-by-step  implemented  systems  engineering 
throughout  the  organization 

•  Is  a  tangible  approach  that  is: 


-  Aimed  at  the  working  level 

-  Affects  all  phases  of  a  program’s  lifecycle 

-  Applicable  throughout  entire  organization 

-  Accounts  for  organization’s  progress  through  metrics 


*  Documented  clearly  in 
Operating  Instructions  (Ols) 


What  Each  01  Has 


•  Brief  and  to  the  point 

•  Pictorially  defined  process  flow 

•  Specific  instructions  for  each  process 
step  aimed  at  working  level 

•  Clearly  outlines  approval  levels 

•  Defines  specific  metrics 

•  States  when/where  show  to  upper 
management 


Tenets  of  Sys  Eng 


•  Our  first-cut  tenet  selection  of 
Systems  Engineering: 

-  Requirements  Management 

-  Risk  Management 

-  Test  Management 

-  Airworthiness 

-  Training 


Tenets  of  Sys  Eng 


•  Our  first-cut  tenet  selection  of 
Systems  Engineering: 

-  Risk  Management 

-  Test  Management 

-  Airworthiness 

-  Training 


Requirements  Mngt  Process  Flowchart 


Receive  Approved 
And  Funded 
Requirements 
Document 


Build 

Integrated 
Requirements  | 
Team  (IRT) 


Identify  and 
Extract  New 
iRequirementsI 


Fill  out 
jRequirements| 
Correlation 
I  Matrix  (RCM)| 


Build  ■!  Identify 
Metrics  pdOperationC 
HScenariosB 


Identify  and 
Extract 
Derived 
Requirements 


iMaintain  and 
Track  RCM 


Pass  RCM 
to 

Risk  Team 


Pass  RCM 
to 

Test  Team 


Unbiased  H 

Update 

Review 

RCM 

Report 

Metrics 


Document 

Lessons 

Learned 


Project  Engineer 
Program 
Manager 
Chief  Engineer 
IRT 


Requirements  Management  Metric 


Requirements 

Reviewed 

Requirements 

Quantified 

^  Requirements 
Changed 


Total  Requirements  =  Stated  Requirements  +  Derived  Requirements 


Requirements  Growth  Metric 


Tenets  of  Sys  Eng 


•  Our  first-cut  tenet  selection  of 
Systems  Engineering: 

-  Requirements  Management 


-  Test  Management 

-  Airworthiness 

-  Training 


Risk  Management  Process  Flowchart 


I  I  Project  Engineer 
I  I  Program  Manager 
I  I  Flight  Director 
I  I  Chief  Engineer 
□  RMT 

Figure  1 .  Flowchart  for  Risk  Management  Process 


Probability 


Risk  #1  Assessment  Matrix 


Low 


Med 


■ 


Technical  Risk:  If  software  complexity 
increases  on  MCS  then  failure  of 
modifications  could  result. 


Mitigation  Plan: 

•  Contractor  is  currently  Capabilities 
Maturity  Model  Integration  (CMMI) 
software  level  3  certified  and  has  plan 
to  reach  level  5  by  contract  award 

•  Government  will  ensure  contractor 
will  work  with  ground  agencies  to 
ensure  software  is  interoperable 

•  Government  will  follow  disciplined 
requirement  matrix  process  outlined 
in  727  ACSG  Operating  Instruction 
(O.l.)  to  prevent  unplanned 
requirements/complexity  increases  & 
track  via  established  metrics 


Risk  Workshop  Completed 
14  Mar  07 


Risk  Quad  Chart 


Risk  Title 

Risk  Tracking  Number 


Background 

Description  of  prob^. 

•  Item  1 

•  Item  2 

•  Item  3 


Risk 

Color 

Code 


Risk  Mitigation  Plan 

*  Proposed  solution  for  implementation 
and  risk  mitigation. 


0 

X 

Actions  to  Date 


Future  Action 


Date 


Established  Risk  Assessment  Date  1 
Completed  Mitigation  Plan  Date  2 

Completed  details  of  mitigation  Date  3 
incorporation  with  contractor 
Received  effort  impact  (cost  and  Date  4 
schedule) 


Proj.Date 

Contract  Award  for  implementation 

Date  1 

Mitigation  Plan  Completion  (or  Date  2 
any  significant  milestones) 

Etc... 


Probability 


Technical  Risk  Summary 


High 


100% 


0% 


Impact 

Negligible  *  Critical 


OVERALL  TECHNICAL 
RISK  IS  LOW 


Risk  Workshop  Completed  - 
14  Mar  07 


Tenets  of  Sys  Eng 


•  Our  first-cut  tenet  selection  of 
Systems  Engineering: 

-  Requirements  Management 

-  Risk  Management 


-  Airworthiness 

-  Training 


Test  Management  Process  Flowchart 


Integrated 

RCM 

Requirements  j 

Team 

2.2  Planning  Phase 


Define  Test  Requirements 

(See  Checklist)  §  2.2.4 


Update 

RCM 


T 

Review 

Nominate 
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Lessons 
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Integrated 

Learned  • 

Test 

Test  Team  (ITT) 

Database 

Organization 

§2.2.1 

§2.2.2 

§2.2.3 

Create  TES 
or  TEMP 
§  2.2.6 


Integrate 
Schedule  and 
Refine  Cost 
with 

Contractor 

§  2. 2. 6. 5 


If  T-2  mod, 
review  AFMCI 
21-126  and 
Prepare  AFMC 
forms  243  and 
244 
§2.2.7 


2.3  Design  Phase 


Technical 

Reviews 

§2.3.1 


Updated 

RCM 


Updated 

RCM 


Each  Review 


Brief  Metrics 

§  2. 2. 5. 8 


Develop 
Test  Metrics 
§  2.2.5 


Note:  Project  Engineer  will  schedule  periodic  meetings  as 
necessary.  See  §2.2. 1.1 

- VQnV 

_ 

Update  TES 
or  TEMP 

§2.3.3 


|2.3  Execution  Phase 


Test 

Readiness 

Review 

§2.4.1 


Test 

Execution 
§  2.4.2 


i 


Review 
Test  Metrics 
§  2.4.3 


Deficiency 
Correction 
§  2.4.5 


Deficiency 
Review 
§  2.4.4 


Post-Test  Activity  §  2.4.6 


Review  Test 
Report 


Input  Lessons 
Learned 


Test  Requirements  Metric 


Test  Requirements  Metric 


140 


40 

20 

0 


Date  1  Date  2  Date  3  Date  4  TRR 


□  Total  #  of  Requirements  □  Quantified  □#  Verifiable  □  Resource  Assigned 


Test  Risks  Management  Metric 


Date  1  Date  2  Date  3  Date  4  TRR 


□  Low  □  Med  ■  High  H  Closed  /  Mitigated 


-F^  cn 


Deficiency  Metric  Report 


Deficiency  Report  Metric 


■  Open  Cat  1  □  Open  Cat  2  ■  Closed 


Tenets  of  Sys  Eng 


•  Our  first-cut  tenet  selection  of 
Systems  Engineering: 

-  Requirements  Management 

-  Risk  Management 

-  Test  Management 


-  Training 


New  Process  to  Ensure  Airworthiness 


Strengthened  SOW  language,  defined  intent  and 

established  clear  RAA 


Ensured  cert  approach  in  place  before  SRR 

Implemented  control  measures  (metrics)  to  verify 

both  designs  and  parts 


Control 

Point 


Design/Part  Certification  Method 


DESIGN  PARTS 


Design  Certification  Breakout 


Total  Mods 


Field  Approval  mod 


Part  Certification  Breakout 


TBDforSRR 


Tenets  of  Sys  Eng 


•  Our  first-cut  tenet  selection  of 
Systems  Engineering: 

-  Requirements  Management 

-  Risk  Management 

-  Test  Management 

-  Airworthiness 


Workforce  Training  Metric 


Org  A  Training  Progress  (45  People) 
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What’s  Next 


•  Aircraft  Structural  Integrity  Program  (ASIP) 

•  Configuration  Control 

•  Service  Life 

•  Mishaps 

•  Obsolescence 

•  Safety 

•  Incentivizing  contractors 


Summary 


•  Measuring  systems  engineering  can  be  a 
daunting  task 

•  327th  ASW  developed  a  means  to  do  this: 

-  Broke  up  sys  eng  into  its  components 

-  Devised  metrics  for  each  component 

-  Institutionalized  by  codifying  in  Ols 

-  Regularly  brief  to  upper  management 

•  Driving  behavior,  but  takes  time 

•  Have  plans  to  do  more... 


Performance  measures  are  being  implemented, 
driving  behavior  AND  making  a  difference 


ml 


Questions  ? 


Incentivizing  Contractors  Metric 


100%- 
75%- 
50% - 

25%  - 


Goal 


%  of  Contracts  with 
Sys  Eng  Incentives 


Risk  Rating 


Risk  Handling  Plan  -  “Waterfall” 


High 


Medium 


Low 


► 


Time 


Sample:  5  -  Level  Risk  Rating  Chart 


ASSESSMENT  GUIDE 


LIKELIHOOD : 

e 

Level 

What  Is  The  Likelihood 
The  Risk  Will  Happen? 

~o 

d 

a 

Remote 

1 

c 

b 

Unlikely 

r*  & 

b 

c 

Likely 

_i 

a 

d 

Highly  Likely 

e 

Near  Certainty 

1  2  3  4  5 
Consequence 


CONSEQUENCE :  — 

Given  The  Risk  Event  is  Realized,  What  is  the  Magnitude  of  the  Impact? 


Level 


Technical 

Performance 


and/or 


Schedule 


and/or 


Cost 


1 

2 


Minimal  or  no  impact  Minimal  or  no  impact 


Acceptable  with  some 
reduction  in  margin 


Additional  resources  required; 
able  to  meet  need  dates 


Minimal  or  no  im 
<5% 


Acceptable  with 
significant  reduction 
in  margin 


Minor  slip  in  key  milestone; 
not  able  to  meet  need  dates 


5  -  7% 


Acceptable,  no 
remaining  margin 


Major  slip  in  key  milestone 
or  critical  path  impacted 


>7-10% 


RISK  ASSESSMENT 


HIGH  -  Unacceptable. 

Major  disruption  likely. 
Different  approach  required. 
Priority  management 
attention  required. 

■  MODERATE  -  Some 
disruption.  Different 
approach  may  be  required. 
Additional  management 
attention  may  be  needed. 

■  LOW  -  Minimum  impact. 
Minimum  oversight  needed 
to  ensure  risk  remains  low. 


and/or  Impact  on  Other  Teams 

pact  None 

Some  impact 


Moderate  impact 


Major  impact 


5  Unacceptable 


Can’t  achieve  key  team  or 
major  program  milestone 


>  10% 


Unacceptable 


Major  Modification  Programs 
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17  Current  Programs  m 
KC-10  AMP  -  ASC  Lead  (ACAT  II) 

KC-10  Dual  406  MHz  ELT  Upgrade  (ACAT  III)* 

KC-10  Iridium  Phone  (ACAT  III)* 

KC-10  UHF  SATCOM  Antenna  (ACAT  III)* 

VC-25  Forward  Lower  Lobe  (FLL)  Cooling  (ACAT  III) 

VC-25  Presidential  Data  System  (PDS)  (ACAT  III)* 

VC-25  C NS/ATM  (ACAT  III)* 

C-20  Gulfstream  Test  Vehicle  (GTV)  (ACAT  III)* 

E-9  Telemetry  Sys  Upgrade  (ACAT  III)* 

E-4B  Mod  Block  I  (ACAT  II)  * 

E-4B  256  Kbps  High  Speed  Data  via  INMARSAT  (ACAT  III)* 
C-12  EFIS  (ACAT  III) 

HFGCS  Network  Control  Station  -  West  (ACAT  III)* 

HFGCS  AFSPC  Test  Range  HF  Modernization  (ACAT  III)* 
HFGCS  Network  Optimization  -  Spiral  II  (ACAT  III)* 

HFGCS  Navy  Consolidation  (ACAT  III)* 

HFGCS  Audit  Log  Upgrade  (ACAT  III)* 


$7.1  M 


$1.03B 
$2.4M 
$2.7M 
$2.6M 
$14. 4M 
$223. 3M 
$41 .8M 
$8.7M 
$5.9M 
$421 ,4M 
$8.4M 
$77. 7M 
$23. 2M 
$3.9M 

$6.4M 

$189K 


Systems  and  Software  Engineering 


Development  and  Validation  of 
a  Systems  Engineering 
Competency  Model 

Don  Gelosh,  Ph.D.,  CSEP-Acq 
Senior  Systems  Engineer 
Systems  Engineering  Support  Office 
Enterprise  Development/  Systems  and  Software  Engineering 
Office  of  the  Deputy  Under  Secretary  of  Defense  (A&T) 


23  October  2008 


Overview 


•  Why  Competency  Management? 

•  Senior  Leadership  Support 

•  Competency  Management  Process 

•  Proposed  Next  Steps 

•  Summary 
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Why  Competency  Management  for  AT&L 
and  Systems  Engineering? 


Systems  and  Software  Engineering 


Competencies  are  observable,  measurable  patterns  of 
knowledge,  skills,  abilities,  behaviors  and  other 
characteristics  that  an  individual  needs  to  perform  work 
roles  or  occupational  functions  successfully. 

Competency  management  helps: 

-  Assess  and  refine  the  requisite  competencies  within  the  current 
workforce 

-  Develop  appropriate  strategies  to  shape  the  skill  sets  and 
capabilities  needed  by  the  future  workforce 

-  I  dentify  overall  capabilities  we  need  to  execute  the  acquisition 
mission 

-  Evaluate  which  competencies  are  mission  critical  and  highest  priority 

-  Develop  solutions  that  will  help  us  mitigate  risk  and  respond  to  the 
challenges 
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Competency  Mode /  Applications 


Systems  and  Software  Engineering 


Agile  Mission  Support 


Enables  tactical,  agile  targeting  of 
resources  to  achieve  desired  ca^fbility 
Enables  improved  organizatior 
refinements  to  align  the  skills  vith 
mission  needs 


Improved  Learnir  g/Training 


Improved  alignment  of  training  to 
“successful  perform  mce”  needs 
Improved  training  in  'estment 
Enables  21st  Century  Training 
Framework  (Core  Pli  s) 


Succession  Planning 


Identify  expected  critical  vacan 
Identify  employees  &  candidate 


High(er)-Performing  Workforce 


Improved  cuyagemem  of  ,.*"'>rkforce  to 
successful  performance”  suppo 
resources  (that  make  a  difference) 
Better  migration  of  Best  Practices 


Human 

.  [ 
.  [ 

Learning 

Resources 

.  [ 

Management 

System 

System 

HR-XML  HB=XMlj=ie_xml  H  R-XM  L 


HR-XML  H 
HR-XML  hi 
HR-XML  ll 


Competency 

Models 


HR-XML 


HR-XML 

HR-XML 

HR-XML 

HR-XML 


HR-XML  HR-XML 

H 

R-XML  HR-XML 

Performance 

H 

Learning 

Management 

F 

Content 

System 

Hk-aivil  mk-aivil 

F 

.  h 

System 

IK- A  ML  MK-AIVIL 

ncios 
te  gape 


Development  &  Career  Pluming 


Enhance  Organization  Development 


Improved  Gap  Assessment  ROI 


Assess  proficiency  AND 
Assess  Mission  Criticality, 
Frequency,  and  Difficulty 
igrate  best  practices  &  tools  for 
cessful  performance 


Stratec^ic  Workforce  Planning 

•  Strategic  p 

•  Enhanced  1 
Mission  Crit 

•  Deliberate, 

•  Information 

anning  enabler  for  leaders 
/lanagement  of 
cal  Competencies 
iarlier  “change  management” 
for  tactical  resource  decisions 

Recruiting  &  Selection 


lmproy§  identification  of  key  behaviors 
controuting  to  successful  performance 
ove  the  “Benefits  Package”  story  - 
orld-class  tools  for  your  development 
and  success” 
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Senior  Leadership  Support  is 

Critical!!! 


Systems  and  Software  Engineering 


The 

President’s 

Management 

Agenda 


Align  with  Senior  Leadership 


National  Security 
Strategy 


•Human  Capital 


•Competitive  sourcing 
•Financial  management 
•Expanded  e-Government 
•Budget  &  Performance  Integration 


•Transform  our  Military  Forces 
•  Implement  QDR 


Robert  Gates  - 
SECDEF 


Gordon  England 
- DEPSECDEF 


Hon.  John  J.  Young,  Jr. 
USD(AT&L) ) 


AT&L 

Strategic 

Thrusts 


Hon.  James  I.  Hon.  Jack  Bell 
Finley  DUSD  (A&T)  DUSD  (L&MR) 


National  Defense  Strategy 


DoD  Alignment 

“The  department  must  have  a  vision 
that  conveys  to  the  public  a 
commitment  to  attract  &  develop  the 
best  mix  of  people,  both  military  & 
civilian.  This  vision  must  be 
supported  by  an  effective  human 
capital  strategy  that  is  actively 
measured  against  well  defined  goals.” 


National  Military  Strategy 


ig  I 

processes  within  the  Dept  to  take 
advantage  of  IT 
Foster  a  culture  of  innovation 
Divest  &  invest  for  the  longer  term 


Quadre  nnial  Defense  Review 

© 

DoD  Civilian  Human  Capital 
Strategic  Plan 


The  AT&L 
Source  Document 


•Continuous  Transformation 
•Capabilities-based 
Approach 
>  Focused  Logistics 
-Joint  Systems 


Big  A”  Acquisition 
Governance 

Risk-based  Source  Selection  & 
•Joint  Systems  Time  Certain  Acquisition 

•  Network  Centric  Operations  Programs 

-  Defense  Human  Capital  Strategy 
•  Competencies  & 
Performance  Criteria 


Strategic  Thrust  #3 
Take  Care  of  Our  People 


T 


AT&L  Workforce  Programs  & 
Initiatives 


AT&L  Competency  Management  Initiative  ...  Enabling  Successful  Acquisition  Outcomes 


A  T&L  Competency  Management  Process 


Collect 

Existing 

Competency 

Data 


I 


Framework 

Development 


Phase  I  -  Convene  an 
expert  panel  (EP) 

Actions: 

•  Develop  a  competency 
framework  &  input  model 

•  EP  identifies  Subject  Matter 
Experts  (SMEs) 

•  EP  communicates 
competency  effort  to  the 
SMEs 

•  Develop  communications 
package 

Goal: 

•  Establish  baseline  of 
existing  competency  model. 

•  Communicate  effort 

Products: 

•  FA  provides  list  of  targeted 
high-performing  SMEs 

•  Obtains  expert  panel 
concurrence  on  baseline 
competency  framework 

•  Obtain  approval  from  Dir, 
HCI  and  FA  on  competency 
model  input 


T 


Approved  Input 
Competency  Model 


Model  Development 


Phase  II  -  Develop  the 
model 

Actions: 

•  SMEs  review  the 
competency  framework 
and  provide  essential  job 
data  through  structured 
interviews  and  online  data 
collection  tools. 

•  SMEs  engaged  to  identify 
key  “work”  situations  and 
competencies  contributing 
to  successful  performance 

•  Analyze  results  and 
develop  competency  model 
content 

Goal: 

•  Model  development  and 
identification  of  key 
behaviors 

Products: 

•  Deliver  Proposed  Model 
Report  to  Dir,  HCI  and  FA 
for  review 


T 


Model  Testing  & 
Refinement 


Phase  III  -  Perform  a  beta 


test  &  refine  model 

Actions: 

•  Collect  and  synthesize 
feedback  from  proposed 
model  report 

•  Pre-assessment 
communications  to 
workforce 

•  Identify  stratified 
workforce  sample 

Goal: 

•  Further  refine  model  to 
include  input  from 
functional  leads 

•  Obtain  FA  and  Dir,  HCI 
approval  for  validation 
assessment 

Products: 

•  Obtain  concurrence  from 
FIPT  on  competency 
model 

•  Obtain  approval  from  Dir, 
HCI  and  FA  on 
competency  model 


I 


Approved  Initial 
Competency  Model  V0.5 


Phase  IV  -  Validate  and 
Assess 

Actions: 

•  Launch  competency 
assessment  tool 

•  Analyze  results  to  evaluate 
model  validity  and 
generalizability  to  the 
workforce 

Goal: 

•  Identify  competencies 
required  for  superior 
performance 

•  Evaluate  proficiency  gaps 
for  validated  competencies 

•  Plan  for  continual  updates 
and  use  of  competency 
model 

Products: 

•  Deliver  proven  (validated) 
competency  model  in  HR 
XML  format 

•  Provide  competency 
validation  and  assessment 
and  obtain  Dir,  HCI  and  FA 
approval 


T 


I 


Competency 
Validation 
&  Assessment 
Report 


Slide  6 


Systems  and  Software  Engineering 


Phase  //  Expert  Panel  and  Competency 
Model  Framework  Development 


AT&L  Systems 
Engineering 
Learning 
Outcomes 


Professional 

Competencies 


Competency 
Model 
Framework 
(40  Technical 
10  Professional) 
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Competency  Mode!  Example 


Systems  and  Software  Engineering 


Unit  of  Competence 
Riding  a  Bicycle 


Competency  1 
Mount  the 
Bicycle 


Element  1  - 
Position  the  Peddle 
Element  2  - 
Swing  leg/Take  seat 
Element  3  - 
Transition  to  Motion 


Competency  2 
Dismount  the 
Bicycle 


Element  1  - 
Slow  Down 
Element  2  - 
Support  at  Stop 
Element  3  - 
Swing  Leg  to  Ground 


Competency  3 
Pedal  the 
Bicycle 


Element  1  - 
Maintain  Balance 
Element  2  - 
Peddle  Fast 
Element  3  - 
Peddle  Slow 


Competency  4 
Maintain  the 
Bicycle 


Element  1  - 
Tire  Pressure 
Element  2  - 
Brake  Operation 
Element  3  - 
Wheel  Balance 
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SE  Competency  Mode, f  Framework 


Systems  and  Software  Engineering 


Technical  Competencies 

Analytical 

SE  Tools  Techniques  Design  Considerations 

Technical  Management 

Technical  Management  Processes 

General 

Total  Systems  Fi'ew 

i 

Technical  Basis  for  Cost 

21 

Decision  Analysis 

36 

Acquisition 

2 

Systems  Engineering  Plans 

22 

Technical  Planning 

37 

IPPD 

3 

Work  Breakdown  Structure 

23 

Technical  Assessment 

38 

Leadership 

4 

Value  Engineering 

24 

Requirements  Management 

39 

International  Acquisition 

5 

Technical  Performance  Measurement 

25 

Risk  Management 

40 

Professional  Ethics 

6 

Trade  Studies 

26 

Configuration  Management 

7 

Modeling  and  Simulation 

27 

Technical  Data  Management 

3 

Failure,  Modes,  Effects  &  Criticality  Analysis 

28 

Interface  Management 

9 

Requirements  Traceability  Matrix 

29 

Technical  Data  Packages 

Professional  Competencies 

10 

Safety  Analysis 

30 

Specifications 

11 

SE  Design  Considerations 

31 

Earned  Value  Management 

41 

Communication 

12 

Requirements  Development 

32 

IMP/IMS 

42 

Analytical  Skills 

13 

Logical  Analysis 

33 

Technical  Reviews 

43 

Decision  Making 

14 

Design  Solution 

34 

Software  Engineering 

44 

Problem  Solving 

15 

Implementation 

35 

Systems  Engineering  by  Phases 

45 

Technology  Management 

16 

Integration 

46 

Team  Building 

17 

Verification 

47 

Influencing  and  Negotiating 

18 

Validation 

48 

Interpersonal  Skills 

19 

Transition 

49 

Strategic  Thinking 

20 

System  Assurance 

50 

Understanding  Attributes  of 
Evidence  and  Rational  Decisions 
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Unit  of 
Competence 

#1  Analytical 

#1  Analytical 

#1  Analytical 

#1  Analytical 

#1  Analytical 


SE  Competency  Model  Examples 


Systems  and  Software  Engineering 


Competency  Elements 


Knowledge  Items 


Technical  Basis  for  Cost  Apply  knowledge  of  cost  drivers  to  Knowledge  of  cost  drivers  and 

develop  cost  estimates  and  program  cost  estimating  techniques  and 

budgets  that  reflect  program  phase  best  practices 

requirements  and  best  practices. 


Systems  Engineering  Plans  Identify  the  proper  points  within  a  Knowledge  of  SEP  preparaton 

program's  lifecycle  to  generate  a  Systems  guidance 
Engineering  Plan  (SEP)  that  describes  the 
program's  SE  processes,  resources, 
metrics,  and  technical  review  process. 


Requirements  Development  Apply  the  Requirements  Development  Knowledge  of  requirements 

process  to  translate  inputs  from  relevant  management  tools 
stakeholders  into  technical  requirements. 


Verification 


Validation 


Apply  the  Verification  process  to  confirm  Knowledge  of  verification  (test 

that  the  system  element  meets  the  design  and  evaluation)  techniques 

specifications  as  defined  in  the  functional, 

allocated,  and  product  baselines  and  to 

answer  the  question:  'Did  you  build  it 

right?' 


Apply  the  Validation  process  to  test  the  Knowledge  of  validation 
performance  of  systems  within  their  (operational  test  and  evaluation) 

intended  operational  environment  and  to  techniques 
answer  the  question  'Did  you  build  the 
right  thing?' 
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Phase  II:  Subject  Matter  Expert  (SME) 

Validation 


Systems  and  Software  Engineering 


•  SMEs  review  the  competency  model  framework  and  provide 
essential  job  data  through  an  online  data  collection  tool. 

•  SMEs  can  add/delete  competencies  and  associated  elements 
and  knowledge  items. 

•  SMEs  must  identify  at  least  two  key  “work”  situations  and 
associated  competencies  that  contribute  to  successful 
performance. 

•  Results  are  analyzed  and  used  to  develop 
a  complete  competency  model. 
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Systems  and  Software  Engineering 


SMEs  review  each 
competency 
element  and 
provide 

information  on: 

*  Frequency 

*  Importance 

*  Level  First  Used 


SME  Competency  Review 


Q  *  o  a  '  LI  a  * 


Unit  of  Competence  #1  Analytical 

Includes  the  analytical  and  technical  processes  of  systems  engineering  with  a  full  understanding  of  tools  and 
techniques  and  all  design  considerations. 


Competency  Element 

Frequency 

Importance 

Level  First  Used 

Technical  Basis  for  Cost  -  Element  1.  Apply 
knowledge  of  cost  drivers  to  develop  cost 
estimates  and  program  budgets  that  reflect 
program  phase  requirements  and  best 
practices. 

O  1  Never 

O  2  Sometimes 

O  3  Often 

O  4  Frequently 

O  5  Very  Frequently 
O  N/A 

O  1  Not  Important 

O  2  Less  Important 

O  3  Moderately  Important 
O  4  Important 

O  5  Very  Important 

O  N/A 

O  1  Entry  Level 

O  2  Mid-Level 

O  3  Expert/Senior  Level 
O  N/A 

Systems  Engineering  Plans  -  Element  1  of 

3  -  Element  2.  Identify  the  proper  points 
within  a  program's  lifecycle  to  generate  a 
Systems  Engineering  Plan  (SEP)  that 
describes  the  program's  SE  processes, 
resources,  metrics,  and  technical  review 
process. 

O  1  Never 

O  2  Sometimes 

O  3  Often 

O  4  Frequently 

O  5  Very  Frequently 
O  N/A 

O  1  Not  Important 

O  2  Less  Important 

O  3  Moderately  Important 
O  4  Important 

O  5  Very  Important 

O  N/A 

O  1  Entry  Level 

O  2  Mid-Level 

O  3  Expert/Senior  Level 
O  N/A 

Systems  Engineering  Plans  -  Element  2  of 

3  -  Element  3.  Develop  the  critical  contents 
of  a  SEP  including  government  and 
contractor  SE  processes,  the  technical 
baseline  approach,  program  control  tools, 
and  the  role  of  SE  to  guide  all  technical 
aspects  of  an  acquisition  program. 

O  1  Never 

O  2  Sometimes 

O  3  Often 

O  4  Frequently 

O  5  Very  Frequently 
O  N/A 

O  1  Not  Important 

O  2  Less  Important 

O  3  Moderately  Important 
O  4  Important 

O  5  Very  Important 

O  N/A 

O  1  Entry  Level 

O  2  Mid-Level 

O  3  Expert/Senior  Level 
O  N/A 

Systems  Engineering  Plans  -  Element  3  of 

3  -  Element  4.  Determine  what  enterprise, 
system  and  software  architectures  are 
needed  to  reason  about  the  system,  to 
inform  recommendations  and  decisions 
regarding  software  implementations  in  the 
context  of  the  system  being  acquired  and  to 
allow  effective  communication  across  the 
stakeholders  throughout  the  system  life 
cycle. 

O  1  Never 

O  2  Sometimes 

O  3  Often 

O  4  Frequently 

O  5  Very  Frequently 
O  N/A 

O  1  Not  Important 

O  2  Less  Important 

O  3  Moderately  Important 
O  4  Important 

C  5  Very  Important 

O  N/A 

O  1  Entry  Level 

O  2  Mid-Level 

O  3  Expert/Senior  Level 
O  N/A 

Work  Breakdown  Structure  -  Element  5. 
Translate  the  system  design  (including  all 
products  and  services)  into  a  Work 

Breakdown  Structure  (WBS)  to  ensure  that 
all  of  the  appropriate  SE  activities  are 
implemented. 

C  i  Never 

C  2  Sometimes 

O  3  Often 

O  4  Frequently 

O  5  Very  Frequently 

O  1  Not  Important 

O  2  Less  Important 

O  3  Moderately  Important 
O  4  Important 

O  5  Very  Important 

O  1  Entry  Level 

O  2  Mid-Level 

O  3  Expert/Senior  Level 
O  N/A 
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Key  Situation  Interviews 


Systems  and  Software  Engineering 


>Key  Situations:  a  method  of  data  collection  from  subject 
matter  experts  regarding  “what  it  takes”  to  perform 
effectively  on  your  job. 

>  Using  the  STARR  Method  of  Description 


Situation/Task  Action 


What  was  the 

What  did  you  do? 

situation  or 

What  were  the 

context?  What 

r-\ 

steps  you  took  to 

were  you  doing? 

get  to  that 

What  task  were 

effective 

you  working  on? 

outcome? 

Reasoning 


What  was  the 
reasoning/ 
rationale  that 
led  to  the 
action? 


Results 


What  was  the 
result/ 
outcome  of 
the  key 
situation? 
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Additional  SME  Questions 


Systems  and  Software  Engineering 


1 .  Do  you  identify  yourself  to  others  as  a  systems  engineer? 

2.  Do  you  have  the  appropriate  resources  to  do  your  job? 

3.  Are  you  allowed  to  apply  new  skills  acquired  through  recent 
education  and  training  to  perform  your  job? 

4.  Does  your  organizational  culture  encourage  the  application 
of  new  skills? 

5.  Do  you  believe  additional  advanced  or  senior  level  training 
in  systems  engineering  is  needed? 

6.  Have  you  received  training  associated  with  integrating 
software  into  warfare  related  systems? 

7.  If  you  answered  yes  to  Question  6,  has  this  training  provided 
you  with  an  adequate  understanding  of  potential  issues 
associated  with  integrating  software  into  warfare  related 
systems? 

8.  What  do  you  see  as  the  primary  community  wide  SPRDE 
workforce  capability  challenge? 


Phase  III:  Test  and  Refine  the  Model  ('Mg) 

Systems  and  Software  Engineering 

•  Collect  and  synthesize  feedback,  refine  the  model. 

•  Further  refine  model  to  include  input  from  Expert  Panel  and 
functional  leads. 

•  Send  pre-assessment  communications  to  workforce. 

•  Identify  stratified  workforce  sample. 


Phase  IV:  Workforce  Assessment 


Systems  and  Software  Engineering 


Launch  competency  assessment  tool. 

Analyze  results  to  evaluate  model  validity  and  general 
applicability  to  the  workforce. 

Identify  competencies  required  for  superior  performance. 
Evaluate  proficiency  gaps  for  validated  competencies. 
Plan  for  continual  updates  and  use  of  competency  model. 


Slide  16 


Proposed  Next  Steps 


Systems  and  Software  Engineering 


Improve  the  Competency  Model: 

•  Compare  and  contrast  with  other  competency  models  - 
leverage  best  of  the  best 

•  Incorporate  results  from  SE  education  and  research  efforts 

•  Develop  a  sub-set  of  “Core  SE  Competencies”  that  define  the 
true  Systems  Engineers 

Apply  the  Competency  Model: 

•  Use  the  Core  Competency  sub-set  to  help  identify  the  true  SEs 
in  the  SPRDE  career  field 

•  Use  the  model  to  develop  criteria  for  hiring  Entry-level, 
Journeyman-level,  and  Highly  Qualified  Experts 

•  Use  the  model  to  drive  SE  education,  training,  and  experience 
opportunities  -  a  guide  to  where  you  should  apply  resources 
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Summary 


Systems  and  Software  Engineering 


To  successfully  develop  and  Implement  a  competency 
management  program,  you  should: 

1.  Develop  a  competency  management  plan. 

2.  Solicit  and  obtain  senior  leadership  support. 

3.  Develop  a  competency  assessment  model  framework. 

4.  Validate  the  model  with  high-performing  subject  matter  experts. 

5.  Test  and  refine  the  model  with  input  from  the  functional  leaders. 

6.  Assess  the  target  workforce  against  the  competency  model  to 
identify  competencies  required  for  superior  performance  and  to 
evaluate  proficiency  gaps. 

7.  Update  the  plan  and  apply  the  competency  model  as  needed. 

8.  Provide  reports. 
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Questions? 

Don  Gelosh,  Ph.D.,  CSEP-Acq 

Senior  Systems  Engineer 
OSD  (AT&L)  SSE/ED 

1851  SOUTH  BELL  STREET 
CRYSTAL  MALL  3  SUITE  102 
ARLINGTON,  VA  22202 

OFFICE:  703-602-0851  EXT  194 
FAX:  703-602-3560 


Systems  and  Software  Engineering 
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Systems  and  Software  Engineering 


Backup  Slides 
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/  NCOSE  UK  SE  Competencies 


Systems  and  Software  Engineering 


INCOSE  UK  Advisory  Board 
Systems  Engineering  Competencies  Framework 


Systems  Thinking 

Systems  concepts 
Super-system  capability  issues 
Enterprise  and  technology 
environment 


Systems  Engineering  Management 

Concurrent  engineering 
Enterprise  Integration 
Integration  of  specialisms 
Lifecycle  process  definition 
Planning,  monitoring  and  controlling 


Holistic  Lifecycle  view 

Determine  and  manage 
stakeholder  requirements 
System  Design: 

Architectural  design 
Concept  generation 
Design  for ... 

Functional  analysis 
Interface  Management 
Maintaining  Design  Integrity 
Modeling  and  Simulation 
Select  Preferred  Solution 
System  Robustness 
Integration  &  Verification 
Validation 

Transition  to  Operation 
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/  NCOSE  SE  Handbook  (33£> 
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Figure  1-1  System  Life  Cycle  Processes  Overview  per  ISO/IEC  15288 
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Overview 


•  Defining  the  Need 

•  SRL  Methodology 

•  Refinement,  Verification  and  Validation 

•  I  implementation  /  Application 


Next  Steps 


The  Complex  System  Development  Problem 


•  A  2006  Government  Accountability  Office  study  of  DOD  technology 
development  practices  concluded: 

-  A  lack  of  insight  into  the  technical  maturity  of  complex  systems  during 
development  has  contributed  to  an  environment  of: 

•  Significant  cost  overruns 

•  Schedule  slips  leading  to  program  delays 

•  Canceled  acquisition  efforts 

__  J_^*™^^™LlAccountability  +  |ntegrjty  *  Reliability 

•  Reduced  system  performance  at  fielding  - - 

•  These  symptoms  will  only  grow  worse  as  demands  for  rapid  development 
and  quick  delivery  increase 

•  DOD  needs  to  strengthen  its  technology  development  monitoring  and  gate 
review  processes 

"Over  the  next  5  years,  many  of  the  programs  in  our  assessment  plan  to  hold  design 
reviews  or  make  a  production  decisions  without  demonstrating  the  level  of  technology 
maturity  that  should  have  been  there  before  the  start  of  development  " 

U.S.  Government  Accountability  Office  on  the  Department  of  Defense,  1999 
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3  GAO  06-883  “Best  Practices:  Stronger  Practices  Needed  to  Improve  DOD  Transition  Process’ 


Defining  Program  Office  Needs 


•  PEO  LMW  /  PMS  420  is  responsible  for  the  development  and 
integration  of  a  series  of  Mission  Modules  to  be  used  on  the  Littoral 
Combat  Ship 

•  Modules  leverage  considerable  amounts  of  technology  from  existing 
programs  of  record  while  also  conducting  new  development 

•  Keys  aspects  of  the  project  include  not  only  monitoring  the  status  of 
technology  development,  but  also  the  maturity  of  the  numerous 
integrations  between  those  technologies 


•  This  has  resulted  in  a  very  complex  and  diverse  system  of  systems 
engineering  activity  with  a  need  to  obtain  quick  and  accurate 
snapshots  of  program  status,  risks,  and  issues 


STEVENS 


Methodology 


TRL  Shortcomings 


•  Application  of  TRL  to  systems  of  technologies  is  not  sufficient  to  give  a 
holistic  picture  of  complex  system  of  systems  readiness 

-  TRL  is  only  a  measure  of  an  individual  technology 

•  Assessments  of  several  technologies  rapidly  becomes  very  complex  without 
a  systematic  method  of  comparison 

•  Multiple  TRLs  do  not  provide  insight  into  integrations  between  technologies 
nor  the  maturity  of  the  resulting  system 

-  Yet  most  complex  systems  fail  at  the  integration  points 


I  ndividual  Technology  System  of  Technologies 


Can  TRL  be  applied?  Can  TRL  be  applied? 

Yes  NO 
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Methodology  Development  Overview 


GOAL:  I  institute  a  robust,  repeatable,  and  agile  method  to  monitor  /  report  system 
development  and  integration  status 


Create  a  System  Readiness  Level  (SRL)  that  utilizes  SME/  developer 
input  on  technology  and  integration  maturity  to  provide  an  objective 
indication  of  complex  system  development  maturity 


S6rj& 


Technology  Readiness 
Levels  (TRL) 


Status  of  technologies 
making  up  the  system 


*xe‘ 


Integration  Readiness 
Levels  (IRL) 


Status  of  connections 
between  the  technologies 


System  Readiness 
Levels  (SRL) 


Overall  system  maturity 
appraisal 


•  Provides  a  system-level  view  of  development  maturity  with  opportunities  to  drill  down 
to  element-level  contributions 

•  Allows  managers  to  evaluate  system  development  in  real-time  and  take  proactive 
measures 

•  Highly  adaptive  to  use  on  a  wide  array  of  system  engineering  development  efforts 

•  Can  be  applied  as  a  predictive  tool  for  technology  insertion  trade  studies  and  analysis 


SRL  Methodology  and  Analysis  Flow 


Step  1 :  Identify  hardware  and 
software  components 


Include  all  technologies  that  make-up 
the  overall  system 


Step  2:  Define  network  diagram 
for  systems 


Step  3:  Define  system 
operational  strings  (If  applicable) 


► 


X 


X 


► 


X 


I 

► 
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Emphasis  is  on  the  proper  depiction  of 
hardware  and  software  integration 
between  the  components 


String  analysis  allows  for  the  option  of 
weighting  the  most  important 
components  and  evaluation  of  alternate 
operational  states 


Step  4  Apply  detailed  TRL  and  IRL 
evaluation  criteria  to  components 
and  integrations 


_ Integration  Maturity  Level  1 

ioo  Components  to  be  integrated  are  selected 
ioo  Component  interface  points  are  identified 
ioo  Documented  phase/build  plan  for  component  availability 


%  1 

Technology  Readiness  Level  1  | 

100 

Physical  laws  and  assumptions  used  in  new  technologies  defined 

50 

Have  some  concept  in  mind  that  may  be  realizable  in  software 

25 

Know  what  software  needs  to  do  in  general  terms 

30 

Paper  studies  confirm  basic  principles  and  system  concepts 

N/A 

Mathematical  formulations  of  concepts  that  might  be  realizable  in  software 

100 

Have  an  idea  that  captures  the  basic  principles  of  a  possible  algorithm 

Basic  scientific  principles  observed 

100 

Research  hypothesis  formulated 

Identify  who  will  perform  research  and  where  it  will  be  done 

60 

Readiness  Level  Percent  Complete  (non-weighted) 

Checklist  style  evaluation  allows  for  the 
ability  to  “take-credit”  for  steps  that  have 
taken  place  beyond  the  current  readiness 
level 
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initial  Architecture  Definition  and  Setup 

Step  5:  Calculate  individual  and 
composite  SRLs 


Step  6:  Document  status  via  roll¬ 
up  charts 


► 


PtftfR  MLL  OA  T  A  K  THEE  TSWFt  IS  MDTWIUL 

System  Detailed  Status 


a 


Input  TRL  and  IRL  evaluations  into 
algorithm  to  compute  an 
assessment  of  overall  system 
status  via  SRLs 


Populate  reporting  chart  templates 
with  evaluation  and  calculation 
outcomes  to  highlight  both  current 
status  and  performance  over  time 


Iterative  SME  Evaluation  Throughout  Development  Cycle 


SRL  Calculation 


•  The  SRL  is  not  user  defined,  but  is  instead  based  on  the  outcomes  of  the 
documented  TRL  and  I RL  evaluations 


Through  mathematically  combining  these  two  separate  readiness  levels,  a 
better  picture  of  overall  complex  system  readiness  is  obtained  by 
examining  all  technologies  in  concert  with  all  of  their  required  integrations 


SRL  a  I  RL  x  TRL 


/• 

r 

1  RLj.1 

IRLi2 

"\ 

irl13 

r 

TRLj 

SRL! 

SRL, 

srl3 

— 

irl12 

IRL22 

IRL.3 

X 

tri_2 

V. 

J 

JRI-13 

IRL.3 

|RL33 

trl3 

C  J 

Composite  SRL  =  1/n  |  SRLj/n  +  SRL^/n  +  SRLj/n 
=  1/n2  SRLi  +  SRL^  +  SRL^ 
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These  values  serve  as  a  decision-making  tool  as  they  provide  a 
prioritization  guide  of  the  system's  technologies  and  integrations  and  point 
out  deficiencies  in  the  maturation  process 


SRL  Calculation  Example 


TRU  =  9 


IRL1j2  =  1 


TRL,  =  6 


TRL  Matrix 


I RL  Matrix 


r 

TRL-l 

r  ^ 

9 

1  RLj 

IRLi2 

I  rl13 

f9 

i 

0^ 

trl2 

— 

6 

irl12 

1  rl2 

|RL23 

— 

l 

9 

7 

trl3 

6 

L  J 

J  RL13 

IRI-23 

|RL3 

L° 

7 

9J 

IRL2,3  =  7 


TRL,  =  6 


SRL  =  I  RL  x  TRL 

(Normalized) 


r 

r 

SRLX  SRL2 

srl3 

J 

^^2 

0.54 

V. 

0.43 

0.59 

J 

Component  SRL*  represents  Technology  “X"  and  its  I  RLs  considered 

Composite  SRL  =  1/  3  (  0.54  +  0.43  +  0.59  )  =  0.52 

The  Composite  SRL  provides  an  overall  assessment  of  the  system  readiness 


Sauser,  B.,  J.  Ramirez- Marquez,  D.  Henry  and  D.  DiMarzio.  (2007).  “A  System  Maturity  Index  for  the  Systems 
10  Engineering  Life  Cycle.”  International  Journal  of  Industrial  and  Systems  Engineering.  3(6).  (forthcoming) 


SRL  Reporting  Method 


SRL  ©  ®  ® 

Concept  Feasibility  Basic 

Definition  Demonstration  Technology 

Integration 


Technology 

Testing 


Tech  1  Tech  3 


G 


System  System  Demo 
Integration  and  Test 


G 

System  to 
System 
Integration 


LEGEND 

|  ~~  1  MP  Technology 
I  |  Sea  Frame  System 

Current  Mission  Package  SRL  Status 
Previous  Mission  Package  SRL  Status 
Current  Mission  System  SRL  Status 
Technology  Readiness  Level 
Integration  Maturity  Level 
System  Readiness  Level  Demarcation 

- Scheduled  Position 

Risk  to  Cost  and/or  Schedule 
Low  Medium  High 


©  ©  & 

Qualification  DT  /  OT  Operational 
Testing  Complete  System  Mission 

Proven 


•  For  complex  systems,  the  amount  of  information  obtained  from  the  SRL 
evaluation  can  be  overwhelming 


•  To  maximize  applicability  SRL  outputs  are  tied  to  key,  program-  specific 
development  milestones 

•  Progress  against  these  milestones  provide  key  insight  to  the  user  regarding 
current  program  status,  risk  and  progress 


STEVENS 


Refinement,  Verification  and  Validation 


"String”  Analysis  I  ncorporated 


Complex  systems  often  offer  numerous  options  for  conducting  operations 


•  Operational  strings  were  created  that  identified  the  components 
required  to  utilize  a  single  function  of  the  system 

•  Assessment  of  the  SRL  for  each  of  these  options  allows  for  a  better 
understanding  of  the  maturity  of  each  operating  configuration 
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Understanding  the  true  status  of  the  system  on  an  operational 
string  level  allows  for  the  opportunity  to  field  initial  capability  earlier 
and  then  add  to  it  as  other  strings  mature 


SRL  Calculators  Developed 


Calculators  are  developed  and  defined  for  the  system  being  evaluated 

Allows  for  real-time  updates  to  TRL  and  I RL  inputs  and  the  resulting  SRL 
evaluation  providing  decision-makers  with  instant  feedback  on  “what  if"  scenarios 

I  ntuitive  interface  removes  the  need  for  the  user  to  manipulate  and  deal  with  the 
mathematics  of  the  SRL  calculation 


Notional  Interface  Diagram 
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Verification  and  Validation  Activities 


I RL  Criteria 

Created  expanded  list  of  I  RL 
criteria  for  each  readiness  level 

Goal  was  to  capture  the  key 
elements  of  the  integration 
maturation  process 

Presented  to  30  integration  SMEs 
from  across  government, 
academia,  and  industry 

Asked  to  assess  importance  of 
each  criterion 

Results  show  solid  buy-in  among 
SMEs  that  identified  criteria  are 
key  factors  in  successful 
integration 


SRL  Evaluation  Process 

•  Conducted  a  “blind  trial"  of  SRL 
methodology  and  evaluation 
process 

•  User's  Guide  and  evaluation 
criteria  were  sent  to  key  system 
SMEs 

•  From  just  these  resources  SMEs 
were  asked  to  conduct  the 
evaluation  and  report  on  the 
results 

•  Compiled  results  and  iterated  on 
lessons  learned  to  improve  the 
process 
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NORTHROP  GRUMMAN 


STEVENS 
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I  mplementation  /  Application 
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Effectively  Channeling  Resources 
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SRL.Q  0  0 


0  0  0  0  0 


Lessons  Learned 


•  Methodology  is  highly  adaptable  and  can  be  quickly  applied  to  a  wide 
variety  of  development  efforts 

•  Programs  tend  to  minimize  the  importance  of  system  and  subsystem 
integration  and  thus  overestimate  the  maturity  of  their  development 

•  Widespread  familiarity  with  TRL  makes  acceptance  and  utilization  of  TRL 
and  I RL  easier 

•  Formulating  the  system  architecture  early  in  development  is  a  key  step  and 
leads  to  an  enhancement  of  the  overall  systems  engineering  effort 

•  System  architecture  formulation  also  provides  the  opportunity  to  bring 
together  SMEs  from  both  the  physical  and  logical  realms  and  necessitates 
insightful  discussions  across  the  team 

•  The  decision  maker  is  afforded  the  ability  to  asses  program  status  from  a 
system  of  systems  perspective 

The  SRL  methodology  delivers  a  holistic  evaluation  of  complex  system 
readiness  that  is  robust  repeatable,  and  agile 


STEVENS 


Next  Steps 


Future  Work  and  Applications 


SRL  methodology  can  be  used  not  only  to  assess  current  program 
performance  against  plan,  but  also  to  roadmap  and  assess  future 

development  options 

Future  work  will  focus  on  the  creation  of  an  interactive  technology 
insertion  options  tradeoff  and  decision  environment 

Key  Aspects: 

•  Development  of  a  tool  to  assess  technology  options  and  architectures 

•  I  ncorporation  of  a  semi- automated  tradeoff  capability  that  considers  SRL, 
cost,  risk,  schedule,  and  performance  impact 

•  Gathering  of  data  from  potential  suppliers  detailing  how  they  fit  into  the 
defined  architecture  and  the  maturity  of  their  product 

Applications: 

•  Future  technology,  obsolescence,  and  upgrade  planning 
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Abstract 


A  2006  Government  Accountability  Office  study  of  Department  of  Defense  (DoD)  technology  transition  processes 
concluded  that  a  lack  of  insight  into  the  technical  maturity  of  complex  systems  during  development  has  lead  to  an 
environment  of  program  cost  overruns,  schedule  slips,  and  reduced  performance.  A  key  aspect  of  current 
development  practices  is  the  reliance  on  the  Technology  Readiness  Level  (TRL)  as  a  core  provider  of  maturity 
assessments.  While  the  TRL  has  been  well  proven  for  its  effectiveness  in  gauging  individual  technology  maturity  in 
research  and  development  applications,  its  extrapolation  to  the  complex  systems  of  systems  integration  dictated  by 
emerging  DoD  requirements  brings  about  a  host  of  issues.  Principally,  by  looking  only  at  the  status  of  individual 
component  technical  maturity,  TRL  fails  to  account  for  the  complexities  involved  in  the  integration  of  these 
components  into  a  functional  system  and  creates  the  opportunity  for  performance  gaps  to  remain  hidden  until  late  in 
the  development  cycle. 


To  address  this  lack  of  a  true  system-level  maturity  analysis  process,  the  Northrop  Grumman  Corporation,  the 
Stevens  Institute  of  Technology,  and  NAVSEA  have  collaborated  to  create  and  implement  a  methodology  known  as 
the  System  Readiness  Level  (SRL).  The  SRL  is  a  composite  rating  system  relying  on  input  from  the  traditional  TRL 
scale  as  well  as  a  new  readiness  gauge  known  as  the  Integration  Maturity  Level  (IRL).  These  two  scales  are 
combined  analytically  to  provide  a  systems  readiness  indicator  that  yields  a  holistic  assessment  of  both  the  maturity 
of  individual  technologies  within  a  system  as  well  as  the  status  of  their  corresponding  integrations  and 
interdependencies.  This  presentation  will  detail  the  application  and  value  of  this  methodology  to  complex  DoD 
integration  efforts  as  well  as  the  theory  behind  the  SRL  concept  and  the  steps  taken  to  minimize  ambiguity  and 
subjectivity  in  the  evaluation  process.  Through  this  it  will  be  shown  that  the  SRL  is  an  effective  tool  for  system 
maturity  and  risk  monitoring  and  contributes  greatly  to  enhancing  development  program  performance  for  complex 
systems. 
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Detailed  SRL  Calculation  Example 

Matrix  Setup 


The  computation  of  the  SRL  is  a  function  of  two  matrices: 

The  TRL  Matrix  provides  a  blueprint  of  the  state  of  the  system  with  respect  to  the  readiness  of 
its  technologies.  That  is,  TRL  is  defined  as  a  vector  with  n  entries  for  which  the  yth  entry 
defines  the  TRL  of  the  yth  technology. 

The  I RL  Matrix  illustrates  how  the  different  technologies  are  integrated  with  each  other  from  a 
system  perspective.  I  RL  is  defined  as  an  nx.n  matrix  for  which  the  element  I RUJ  represents  the 
maturity  of  integration  between  the  /th  and  y  th  technologies. 

Populate  these  matrices  with  the  appropriate  values  from  the  previously  documented  TRL 
and  I  RL  component  evaluations  and  then  normalize  to  a  (0,1)  scale  by  dividing  through 
by  9 

For  an  integration  of  a  technology  to  itself  (e.g.  !RLnn)  a  value  of  "9"  should  be  placed  in 
the  matrix 

For  an  instance  of  no  integration  between  technologies  a  value  of  "0”  should  be  placed  in 
the  matrix 


Mxi  = 

TRLX 

trl2 

[/ml] . = 

~IMLU 

iml21 

IMLn 

iml22 

...  IMLin  ~ 

...  iml2u 

TRL,, 

JMLnl 

iml„2 

...  lMLm_ 
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Detailed  SRL  Calculation  Example 

Calculation 


•  Obtain  an  SRL  matrix  by  finding  the  product  of  the  TRL  and  I RL  matrices 


[SRLlxi  =  [IML\ . x  [TRL\rl 

•  The  SRL  matrix  consists  of  one  element  for  each  of  the  constituent 

technologies  and,  from  an  integration  perspective,  quantifies  the  readiness 
level  of  a  specific  technology  with  respect  to  every  other  technology  in  the 
system  while  also  accounting  for  the  development  state  of  each  technology 
through  TRL  Mathematically,  for  a  system  with  ^technologies,  [SRL]  is: 


SRLX 

" IMLnTRL{  +  IMLnTRL2  + ...  +  IMLlnTRLn  ~ 

[srl]= 

srl2 

— 

IML2lTRLx  +  IML22TRL2  + ...  +  IML2nTRLn 

SRLn_ 

IML.TRL,  +IML2TRL2  +  ...  +  IMLTRLn 

_  n  1  1  nz  Z  nn  n  _ 

Decision  Support  Metrics  for  Developmental  Life  Cycles,  Users  Guide:  Version  2.0,  Northrop  Grumman  Corp. 
28  and  Stevens  Institute  of  Technology,  5  September  2007 


Detailed  SRL  Calculation  Example 

Analysis 


•  Each  of  the  SRL  values  obtained  from  the  previous  calculation  would 
fall  within  the  interval  (0,  #  of  I  ntegrations  for  that  Row).  For 
consistency,  these  values  of  SRL  should  be  divided  by  the  number  of 
integrations  for  that  row  of  the  matrix  to  obtain  the  normalized  value 
between  (0,1).  (e.g.  if  there  are  four  non-zero  numbers  in  the  I RL 
matrix  for  that  row,  divide  by  four) 

•  This  number  should  then  be  multiplied  by  9  to  return  to  the  familiar 
(1,9)  scale 

•  For  Example: 


r 

1  RL, 

irl12 

irl13 

f° 

1 

0^ 

irl12 

1  RI-2 

irl23 

— 

1 

0 

7 

v|RLi3 

lRL23 

IRLV 

1° 

7 

°J 

* 

* 

* 


1  Integration  (Divide  SRL  for  that  Row  by  1  and  multiply  by  9) 

2  Integrations  (Divide  SRL  for  that  Row  by  2  and  multiply  by  9) 
1  Integration  (Divide  SRL  for  that  Row  by  1  and  multiply  by  9) 


Decision  Support  Metrics  for  Developmental  Life  Cycles,  Users  Guide:  Version  2.0,  Northrop  Grumman  Corp. 
29  and  Stevens  Institute  of  Technology,  5  September  2007 


Detailed  SRL  Calculation  Example 

Analysis 


SRL 


SRL , 


SRL 2  SRL j 


•  These  individual  values  serve  as  a  decision-making  tool  as  they  provide 
a  prioritization  guide  of  the  system's  technologies  and  integrations  and 
point  out  deficiencies  in  the  maturation  process 


•  The  composite  SRL  for  the  complete  system  is  the  average  of  all 

normalized  SRL  values.  (Note  that  weights  can  be  incorporated  here  if 
desired.) 


SRL 


Composite 


( SRL,  SRL ,  SRL,  \ 

- '-  + - i  +  ...  + - ^ 

V  n  n  n 


n 


•  A  standard  deviation  can  also  be  calculated  to  indicate  the  variation  in 
the  system  maturity 
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SRL  Calculation  Example 

Normalizing  the  TRLs  and  I  RLs 


IRLj 

irl12 

1  rl13 

r  ~\ 

TRL-l 

irl12 

irl2 

IRL23 

trl2 

J  RL13 

irl23 

irl3^ 

trl3 

^  J 

Populate  with 
Evaluation  Results 


Non- Normalized  [(1,9)  scale] 


r 


v 


9 

1 

0 


1 

9 

7 


0 

7 

9 


Divide  by  9 


Remember. . .  a  technology  integrated  with  itself 
receives  an  iRL  value  of  9  (e.g.  iRLu), 
while  technologies  for  which  there  is  no  connection 
between  them  receive  a  value  ofO  (e.g.  !RL13). 


Normalized  [(0,1)  scale] 


r 

1.0 

0.11 

H 

■ 

O 

J 

0.11 

1.0 

.78 

0.67 

L 

0 

.78 

o 

■ 

H 

0.67 

J 
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SRL  for  System  Alpha 

Calculating  the  SRL  and  Composite  Matrix 


SRL  =  I RL  x  TRL 

Component  SRL 

SRLl  srl2  srl3 


r 


1.07 

1.30 

1.19 

= 

0.54 

0.43 

0.59 

(0,n)  scale 


Where  "n"  is  equal  to  the  number  of 
integrations  for  that  technology 


SRL!  SRL2  SRL3 

Component  SRL^  represents  Technology  "X"  and  its  I  RLs  considered 


(0,1)  scale 


Composite  SRL 

Composite  SRL  =  1/  3  (  0.54  +  0.43  +  0.59  ) 


=  0.52 

The  Composite  SRL  provides  an  overall  assessment  of  the  system  readiness 

Both  individual  and  composite  scores  provide  key  insights  into  the  actual  maturity  of  the 
system  as  well  as  where  risk  may  He  and  attention  directed  for  greatest  benefit 

Sauser,  B.,  J.  Ramirez- Marquez,  D.  Henry  and  D.  DiMarzio.  (2007).  “A  System  Maturity  Index  for  the  Systems 
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NOTE:  ALL  DATA  IN  THIS  TEMPLATE  IS  NOTIONAL 


System  Detailed 


Status 


Composite  Actual 

Planned 

SRL 

SRL  w/o 
Platform  1 
Integrations 

LEGEND 

|  I  Platform  1  System 
[  ]  Platform  2  System 

Current  Composite  SRL  Status 
Previous  Composite  SRL  Status 


o 

o 

o 


V 

V 


Individual  technology  SRL  Status 
Technology  Readiness  Level 
Integration  Maturity  Level 

System  Readiness  Level  Demarcation 

Scheduled  Position 

Low  Risk  to  Cost  and/or  Schedule 

Moderate  Risk  to  Cost  and/or  Schedule 
High  Risk  to  Cost  and/or  Schedule 


Data  Collection  Period:  XX/XX/XX  -  X/XX/XX 
Previous  Report  Date:  XX/XX/XX 
Schedule  Updated:  XX/XX/XX 


Tech  5  Tech  6  Tech  10 
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SRL 


Tech  7 


.Tech  8 


Tech  12  Tech  11  Tech  9  Tech  4 


? 


9 


O 


-O- 


NOTE:  ALL  DATA  IN  THIS  TEMPLATE  IS  NOTIONAL 


Program  Status  Roll-up 


1  FYXX 

FYXX 

FYXX 

FYXX  1 

Q1  Q2  Q3  Q4  Q1  Q2  Q3  Q4  Q1  Q2  Q3  Q4  Q1  Q2  Q3  Q4 


LEGEND 

; 

Current  Reporting  Period  Status 

- Scheduled  Position 

Previous  Reporting  Period  Status 

System  Readiness  Level 

Data  Collection  Period:  XX/XX/XX  -  X/XX/XX 
Previous  Report  Date:  XX/XX/XX 
Schedule  Updated:  XX/XX/XX 


Semantic  Syntactic  Pragmatic 


What  is  an  I RL? 
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A  systematic  measurement  reflecting  the  status  of  an  integration 

connecting  two  particular  technologies 


1  RL 

Definition 

9 

1  ntegration  is  Mission  Proven  through  successful  mission  operations. 

8 

Actual  integration  completed  and  Mission  Qualified  through  test  and  demonstration,  in  the  system  environment. 

7 

The  integration  of  technologies  has  been  Verified  and  Validated  with  sufficient  detail  to  be  actionable. 

6 

The  integrating  technologies  can  Accept,  Translate,  and  Structure  Information  for  its  intended  application. 

5 

There  is  sufficient  Control  between  technologies  necessary  to  establish,  manage,  and  terminate  the  integration. 

4 

There  is  sufficient  detail  in  the  Quality  and  Assurance  of  the  integration  between  technologies. 

3 

There  is  Compatibility  (i.e.  common  language)  between  technologies  to  orderly  and  efficiently  integrate  and  interact. 

2 

There  is  some  level  of  specificity  to  characterize  the  1  nteraction  (i.e.  ability  to  influence)  between  technologies  through 
their  interface. 

1 

An  Interface  between  technologies  has  been  identified  with  sufficient  detail  to  allow  characterization  of  the  relationship. 

Gove,  R.  (2007)  Development  of  an  Integration  Ontology  for  Systems  Operational  Effectiveness.  M.S.  Thesis. 
Stevens  Institute  of  Technology.  Hoboken,  NJ 


SRL  Algorithm  Sensitivity  Evaluated 


•  Observed  that  the  SRL  algorithm  did  not  take  into  account  the  varying 
levels  of  “importance"  between  technologies 

•  Examined  the  sensitivity  of  the  algorithms  to  changes  in  the  TRL  and  I RL 
ratings  of  systems  with  varying  levels  of  importance 


•  Modified  the  methodology  to  automatically  include  weightings  for  those 
technologies  that  are  most  important  by  looking  at  operational  "strings"  or 
mission  threads 


37 


SRL  Response  Analysis 

IML  =  1  *  Indicates  unreasonable  combination  IML  =  4 


Components  to  be  integrated  are  selected  and 
interfaces  identified 


Integration  and  data  requirements  are  defined; 
low  fidelity  experimentation 


TRL 

Composite  SRL 

1 

0.06 

3 

0.17 

5 

0.28 

7 

0.39 

9 

0.51* 

IML  =  7 

End-to-end  system  integration  accomplished; 
_ prototype  demonstrated _ 


TRL 

Composite  SRL 

1 

0.10* 

3 

0.29* 

5 

0.49 

7 

0.68 

9 

0.88 

TRL 

Composite  SRL 

1 

0.08 

3 

0.23 

5 

0.38 

7 

0.54 

9 

0.69* 

IML  =  9 

System  installed  and  deployed  with  mission 
_ proven  operation _ 


TRL 

Composite  SRL 

1 

0.11* 

3 

0.33* 

5 

0.56* 

7 

0.78 

9 

1.00 

Algorithms  Evaluated  for  Sensitivity 


TRL  Variation  Analysis 

All  TRLs  in  the  system  are  set  to  9  with  the  exception  of  the  one 
corresponding  to  the  system  in  each  row,  which  was  set  to  1. 


IRL  Variation  Analysis 


All  IRLs  in  the  system  are  set  to  9  with  the  exception  of  the  one 
corresponding  to  the  link  in  each  row,  which  was  set  to  1 


Standard 

Methodology 

Non-connected, 
Self  1  RLs  =  0 

Sys 

String 

Sys 

String 

MPCE 

6  Connections 

Used  by  all  Threads 

8.6 

7.9 

7.9 

7.2 

Radar 

1  Connections 

Used  by  all  Threads 

8.6 

7.9 

8.8 

8.5 

MH-60S 

7  Connections 

Used  by  5  Threads 

8.6 

8.4 

7.7 

8.1 

COBRA 

1  Connections 

Used  by  1  Thread 

8.6 

8.9 

8.8 

8.9 

Standard 

Methodology 

Non-connected, 
Self  1  RLs  =  0 

Sys 

String 

Sys 

String 

MPCE  -  CMS 

Used  by  all  Threads 

9.0 

8.7 

8.6 

8.0 

Radar  -  CMS 

Used  by  all  Threads 

9.0 

8.7 

8.6 

8.0 

MH-60S  -  MPCE 

Used  by  5  Threads 

9.0 

8.8 

8.6 

8.4 

COBRA- VTUAV 

Used  by  1  Thread 

9.0 

9.0 

8.6 

8.9 

NOTE:  There  are  9  total  threads 


NOTE:  There  are  9  total  threads 


Comparative  Sensitivity  —  A  look  at  how  the  algorithms  penalized  the  SRL  rating  relative  to  one  another  (1  is  most  severe) 


Standard 

Methodology 


Non-connected, 
Self  I  RLs  =  0 


Standard 

Methodology 


Non-connected, 
Self  I  RLs  =  0 


1.)  MPCE 

2.)  MH-60S 

3.)  Radar 

4.)  COBRA 

1,2 


1,2 


Sys 

String 

2 

i 

1 

2 

3,4 

3 

3,4 

4 

1. )  MPCE  -  CMS 

2. )  MH-60S  -  MPCE 

3. )  Radar  -  CMS 

4. )  COBRA  -  VTUAV 
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New  Life  Cycle  Old  Life  Cycle 


Subtle,  But  Substantial  Changes 


Process  entry  at  Milestones  A ,  B,  or  C 
Entrance  criteria  met  before  entering  phase 

Evolutionary  Acquisition  or  Single  Step  to  Full 
Capability 


/a\ /b\' 


(Program 

Initiation) 


IOC 


FOC 


Concept 

Refinement 

Technology 

Development 

System  Development 
&  Demonstration 

Production  & 
Deployment 

Operations  & 
Support 

Concept 

S  Decision 

A  Ses|9n 

<  >  Readiness 
Review 

LRIP/IOT&E  /S  Decision 

>r  Review 

Pre-Systems  Acquisition 


Systems  Acquisition 


Sustainment 


User  Needs 


Technology  Opportunities  &  Resources 


/*\ /b\ 


(Program 

Initiation) 


The  Materiel  Development  Decision  precedes 
entry  into  any  phase  of  the  acquisition  framework 

Entrance  criteria  met  before  entering  phase 

Evolutionary  Acquisition  or  Single  Step  to 
Full  Capability 


/f\ 


IOC 


FOC 


Materiel 

Solution 

Analysis 

Technology 

Development 

Engineering  and 
Manufacturing 
Development  & 
Demonstration 

Production  & 
Deployment 

Operations  & 
Support 

^  Materiel 
>  Development 
^  Decision 

/\Post-CDR 

Assessment 

LRIP/IOT&E  A  Decision 

v  Review 

.Pre-Systems  Acquisition^^^^ 

Decision  Point  /\=  Milestone  Review 


Systems  Acquisition 


/\  Sustainment 


Overview  of  Draft  Acquisition  Policy  Changes* 


❖  Mandatory  Materiel  Development  Decision  (MDD) 

❖  Mandatory  competing  prototypes  before  MS  B 

❖  Mandatory  PDR  and  a  report  to  the  MDA  before  MS  B  (moves  MS  B  to  the  right) 


❖ 


Configuration  Steering  Boards  at  Component  level  to  review  all 
requirements  changes  MSA  MSB  MSC 


Strategic 

Guidance 


Joint 

Concepts 


Materiel 

Solution 


Technology 


Analysis  Development 


CDD 


Engineering  and 
Manufacturing 
Development  and 
Demonstration 


CPD 

/ 


Production  and 
Deployment 

o 


o&s 


JCIDS  Process 


Full  Rate  Production 
Decision  Review 


❖ 


PDR  CDR 

Renewed  emphasis  on  manufacturing  during  system  development: 


•  Re-titles  SDD  phase  to  EMDD  with  two  sub  phases:  Integrated  System 
Design  and  System  Capability  and  Manufacturing  Process  Demonstration 

*  Establishes  consideration  of  manufacturing  maturity  at  key  decision  points 


❖  Mandatory  system-level  CDR  with  an  initial  product  baseline  and  followed  by 
a  Post-CDR  Report  to  the  MDA 

❖  Post-CDR  Assessment  by  the  MDA  between  EMDD  sub  phases 


‘Coordination  Draft,  DoDI  5000.02 


Mandatory  “Materiel  Development  Decision” 


Technology  Opportunities  &  Resources 


Materiel  Development  Decision  precedes  entry  into  any  phase  of 
the  acquisition  framework 

Entrance  criteria  met  before  entering  phase 
Evolutionary  Acquisition  or  Single  Step  to  Full  Capability 


MS  C 


Full  Rate 
Production 
Decision  Review 


Strategic  Joint 
Guidance  Concepts 
(OSD/JCS)  (COCOMs) 


Materiel 

Solution 

Analysis 


Technology 

Development 


Engineering  & 
Manufacturing 
Development  & 
Demonstration 


CPD 

F 


Production  and 
Deployment 


O&S 


JCIDS  Process 


AoA 

_ 


incremental  Development 


> 


“  When  the  ICD  demonstrates  the  need  for  a  materiel  solution,  the  JROC  will  recommend  that  the  MDA 
consider  potential  materiel  solutions.  The  MDA,  working  with  appropriate  stakeholders,  shall  determine 
whether  it  is  appropriate  to  proceed  with  a  Materiel  Development  Decision.  ...  If  the  MDA  decides  that 
additional  analysis  is  required,  a  designated  office  shall  prepare,  and  the  MDA  shall  approve,  study 
guidance  to  ensure  that  necessary  information  is  available  to  support  the  decision.  . . .  The  Materiel 
Solution  Analysis  Phase  begins  with  the  Materiel  Development  Decision  (MDD)  Review.  The  MDD  Review 
is  the  formal  entry  point  into  the  acquisition  process  and  shall  be  mandatory  for  all  programs.  . . .  At  the 
MDD  Review,  the  Joint  Staff  shall  present  the  JROC  recommendations  and  the  DoD  Component  shall 
present  the  ICD  including:  the  preliminary  concept  of  operations,  a  description  of  the  needed  capability, 
the  operational  risk,  and  the  basis  for  determining  that  non-materiel  approaches  will  not  sufficiently 
mitigate  the  capability  gap.  The  Director,  PA&E,  shall  propose  study  guidance  for  the  AoA.  . . .  The  MDA 
shall  approve  the  AoA  study  guidance;  determine  the  acquisition  phase  of  entry;  identify  the  initial  review 
milestone;  and  designate  the  lead  DoD  Component(s).  The  MDA  decisions  shall  be  documented  in  an 
Acquisition  Decision  Memorandum  (ADM).  ” 


♦>  ♦>  ♦>  ♦> 


FY08  National  Defense  Authorization  Act 


Mandates  Milestone  A 
approval  prior  to  technology 
development  for  a  major 
weapon  system 

Requires  MDA  Certification 
prior  to  Milestone  A  for 
MDAPs 

Changed  Milestone  B 
Certification  Requirements 

Mandates  reporting  and 
notification  of  program 
cost  changes 
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«*•« .  s,on 


Prototyping  and  Competition 


“Evolutionary  acquisition  requires  . . . 
Technology  development  preceding 
initiation  of  an  increment  shall  continue 
until  the  required  level  of  maturity  is 
achieved,  prototypes  of  the  system  or 
key  system  elements  are  produced,  and 
a  pre  minary  design  is  completed.  . . 

“The  TDS  and  associated  funding  shall 
provide  for  two  or  more  competing 
teams  producing  prototypes  of  the 
system  and/or  key  system  elements 
prior  to,  or  through,  Milestone  B.” 


THE  under  secretary  of  defense 

PCNTAG°N 

WASHINGTON,  DC  20301.3010 


i  *  sep  im 


tMO,WND0Mroi,^S 

DIRECTORS  Of  THkSSse°^»SS  COMMAm 
SUBJECT:  Prototyping  and  Competition 

inadequate  '^o^^maSy  ' ““  Pro«™»“  were initialed  with 

development  path.  Specificity,  program  of  ““  cri,i“‘ 

Proposal,  that  provided  ^  ^  °«  P-per 

for  estimating  dcvelopmcnr  and  prociuemenfcosii^'^  nSk  “d  a  wesk  foundation 
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&  Campedag  Placing  AotyKs^ev^r'0^  U“’oug''  Mil«'one  (MS) 
mk.  validate  designs,  validate  com  estimte*  tvL  *?  ™  dcment*  wi,J  reduce  technical 
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provide  an  opportunity  to  develop  and  enhance  mtem  '  *C  pro,ol)’Pin* 
programs  provide  a  melllu(J  *Wfc  Third,  the 

e  government  and  our  industrial  base  Fourth  »  10  071101  *°re  engineering  skills 
generation  of  young  scientists  and  engineers  in  \pr“°!-vPe  en®rts  can  attract  a  new 
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Preliminary  Design  Review  Precedes  MS  B 


MSA 

A 

MSB  MSB 

MS  C 

A 

<^>  &!E£Si 

®>  De" 

jineering  &  Manufacturing 
slopment  &  Demonstration 

CPD 

7 

PD  °&S 

CDD 

- 

CDD 

FRPDR 

CHARACTERISTICS 

MS  B  moved  “to  the  right”  to  allow  contractor  preliminary  design  to  inform  requirements, 
estimated  costs,  and  schedule. 

Technology  Development  extended  through  formal  Preliminary  Design  Review  (PDR). 

PROCESS 

Preliminary  design  based  on  DRAFT  CDD  to  facilitate  trades  before  JROC  approval. 

Competitive  environment  sustained  up  to  and  perhaps  through  MS  B.  MDA  conducts  MS  B 
review  as  described  in  current  policy. 

SUPPORTING 

PDR  Report  from  PM. 

INFORMATION 

Current  statutory  and  regulatory  information 

BENEFITS 


♦♦♦  Ties  program  decision  to  event-based  (product-based)  technical  review 
♦♦♦  Most  derived  requirements  surfaced 

♦♦♦  Better  understanding  of  cost,  schedule,  and  performance  risk  when  the  APB  is  approved  and  SAR  reporting  begins 

♦♦♦  Opportunity  for  MDA  to  defer  (in  coordination  with  requirements  authority)  unachievable  requirements  to  next  increment 

♦♦♦  Final  requirements  informed  by  detailed  design 

♦♦♦  Early  indicator  of  manufacturing  and  production  issues 

♦♦♦  Logical  extension  of  prototyping  and  competition  policy 


Preliminary  Design  Review 


§  3.5.1 1.  A  Preliminary  Design  Review  (PDR)  shall  be  conducted 
for  the  candidate  design(s)  to  establish  the  allocated  baseline 
(hardware,  software,  human/support  systems)  and  underlying 
architectures  and  to  define  a  high-confidence  design.  All  system 
elements  (hardware  and  software)  shall  be  at  a  level  of  maturity 
commensurate  with  the  PDR  entrance  and  exit  criteria.  A 
successful  PDR  will  inform  requirements  trades;  improve  cost 
estimation;  and  identify  remaining  design,  integration,  and 
manufacturing  risks.  The  PDR  shall  be  conducted  at  the  system 
level  and  include  user  representatives  and  associated  certification 
authorities.  The  PDR  Report  shall  be  provided  to  the  MDA  at 
Milestone  B  and  include  recommended  requirements  trades  based 
upon  an  assessment  of  cost,  schedule,  and  performance  risk. 


Re-Titled  Engineering  and  Manufacturing 
Development  and  Demonstration  Phase 


MDA  Conducts  Post-CDR  Assessment 


A_ A 

A 

Iqqx  Materiel  Solution 
Analysis 
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CHARACTERISTICS 

/  Critical 

Post-CDR  Assessment  replaces  Design  Readiness  Review.  Design  Review 

PROCESS 

Post-CDR  Assessment  is  a  formal,  Milestone  Decision  Authority  (MDA)-  C 
conducted  decision  event.  PM  describes  product  baseline,  completed  build-to 
packages,  a  summary  of  issues  and  an  assessment  of  program  risk  based 
on  the  CDR  report  and  summarized  EVM  data.  Review  considers  whether, 
based  on  the  Program  Manager’s  report,  the  program  is  able  to  provide 
capability  consistent  with  the  Acquisition  Program  Baseline  approved  at 
Milestone  B.  The  MDA  determines  whether  (1)  an  adjustment  should  be 
made,  or  (2)  the  program  should  be  permitted  to  proceed  without  change. 

SUPPORTING 

INFORMATION 

System-Level  CDR  Report 

BENEFITS 

Capitalizes  on  a  well-defined,  event-based,  technical  review 

Decisions  based  on  enhanced  knowledge  of  program  and  associated  contract,  all  derived  requirements  surfaced, 

design  uncertainties  resolved,  development  and  production  costs  well  defined 

Opportunity  for  MDA  to  assess  design  maturity,  e.g.,  drawings  complete 

May  provide  opportunity  to  update  “current”  baseline  if  consistent  with  statute  (“re-structure”) 

An  opportunity  to  defer  “derived”  requirements  if  inconsistent  with  cost  /  schedule  thresholds 


Post-CDR  Assessment 


§3. 6.4.2.  Post-Critical  Design  Review  (CDR)  Assessment  The 
MDA  shall  conduct  a  formal  program  assessment  following  system- 
level  CDR.  The  system-level  CDR,  which  shall  be  conducted  as  soon 
as  practicable  after  program  initiation,  provides  an  opportunity  to 
assess  design  maturity  as  evidenced  by  measures  such  as:  successful 
completion  of  subsystem  CDRs;  the  percentage  of  hardware  and 
software  product  build-to  specifications  and  drawings  completed  and 
under  configuration  management;  planned  corrective  actions  to 
hardware/software  deficiencies;  adequate  developmental  testing;  an 
assessment  of  environment,  safety  and  occupational  health  risks; 
a  completed  failure  modes  and  effects  analysis;  the  identification  of 
key  system  characteristics,  manufacturing  feasibility,  and  critical 
manufacturing  processes;  an  estimate  of  system  reliability  based  on 
demonstrated  reliability  rates;  etc. 


Post-CDR  Report 


§  3. 6 A2A.  The  PM  shall  provide  a  Post-CDR  Report  to  the  MDA  that  provides 
an  overall  assessment  of  design  maturity  and  a  summary  of  the  system-level 
CDR  results  which  shall  include,  but  not  be  limited  to: 


§  3. 6.4.2. 1.1.  The  names,  organizations,  and  areas  of  expertise  of 
independent  subject  matter  expert  participants  and  CDR  chair; 

§  3. 6.4.2. 1.2.  A  description  of  the  product  baseline  for  the  system  and  the 
percentage  of  build-to  packages  completed  for  this  baseline; 

§  3. 6.4.2. 1.3.  A  summary  of  the  issues  and  actions  identified  at  the  review 
together  with  their  closure  plans; 

§  3. 6.4.2. 1.4.  An  assessment  of  risk  by  the  participants  against  the  exit 
criteria  for  the  EMDD  Phase;  and 

§  3. 6.4.2. 1.5.  Identification  of  those  issues/risks  that  could  result  in  a 
breach  to  the  program  baseline  or  substantively  impact  cost,  schedule,  or 
performance. 

§  3. 6.4. 2.2.  The  MDA  shall  review  the  Post-CDR  Report  and  the  PM's 
resolution/mitigation  plans  and  determine  whether  additional  action  is  necessary 
to  satisfy  EMDD  Phase  exit  criteria  and  to  achieve  the  program  outcomes 
specified  in  the  APB.  The  results  of  the  MDA's  Post-CDR  Assessment  shall  be 
documented  in  an  ADM. 


Codifies  OSD  SE  Role  in 
Program  Oversight 

§3.9.6.  Program  Support  Reviews  (PSR).  PSRs  are  a  means  to  inform  an  MDA  and 
Program  Office  of  the  status  of  technical  planning  and  management  processes 
by  identifying  cost,  schedule,  and  performance  risk  and  recommendations  to 
mitigate  those  risks.  PSRs  shall  be  conducted  by  cross-functional  and  cross- 
organizational  teams  appropriate  to  the  program  and  situation.  PSRs  for  AC  AT 
ID  and  I  AM  programs  shall  be  planned  by  the  Director,  Systems  and  Software 
Engineering  to  support  OIPT  program  reviews,  at  other  times  as  directed  by  the 
USD  (AT&L),  and  in  response  to  requests  from  PMs. 

Enclosure  5.  §  E5.7.2.  The  DUSD(A&T)  shall  conduct  an  independent  Assessment  of 
Operational  Test  Readiness  (AOTR)  for  all  AC  AT  ID  programs  and  special 
interest  programs  designated  by  the  USD(AT&L).  Each  AOTR  shall  consider 
the  risks  associated  with  the  system's  ability  to  meet  operational  suitability  and 
effectiveness  goals.  This  assessment  shall  be  based  on  capabilities  demonstrated 
in  DT&E,  and  OAs,  and  criteria  described  in  the  TEMP.  The  AOTR  report  shall 
be  provided  to  the  USD(AT&L),  D,OT&E,  and  Component  Acquisition 
Executive  (CAE). 

§  E5.7.3.  The  CAE  shall  consider  the  results  of  the  AOTR  prior  to 
making  a  determination  of  materiel  system  readiness  for  IOT&E. 


New  Systems  Engineering  Enclosure 


Codifies  three  previous  SE  policy  memoranda 
Codifies  a  number  of  SE-related  policies  and 
Statutes  since  2003: 

■  Environment,  Safety,  and  Occupational  Health 

■  Corrosion  Prevention  and  Control 

■  Modular  Open  Systems  Approach 

■  Data  Management  and  Technical  Data  Rights 

■  Item  Unique  Identification 

■  Reliability,  Availability,  and  Maintainability 

Introduces  new  policy  on  Configuration  Management 


Enclosure  12.  Systems  Engineering 


E12.1.  Systems  Engineering  Across  the  Acquisition  Lifecycle. 

E12.2.  Systems  Engineering  Plan  (SEP). 

E12.2.1.  PMs  shall  prepare  a  SEP  for  each  milestone  review,  beginning  with 
Milestone  A.  At  Milestone  A,  the  SEP  shall  support  the  TDS;  at  Milestone  B  or 
later,  the  SEP  shall  support  the  Acquisition  Strategy. 

E12.2.2.  The  DUSD  (A&T)  shall  be  the  SEP  approval  authority  for  programs 
that  will  be  reviewed  by  the  DAB/ITAB. 

El 2.3.  Systems  Engineering  Leadership.  Each  PEO,  or  equivalent,  shall  have  a 
lead  or  chief  systems  engineer  on  his  or  her  staff  responsible  to  the  PEO  for 
systems  engineering  across  the  PEO’s  portfolio  of  programs.  . . .  and  shall: 

E12.3.1.  Review  assigned  programs’  SEPs  and  oversee  their  implementation. 

El 2. 3. 2.  Assess  performance  of  subordinate  lead  or  chief  system  engineers  ... 

E12.4.  Technical  Reviews.  Technical  reviews  shall  be  event  driven,  conducted 
when  documented  entrance  criteria  are  met,  and  include  participation  by  subject 
matter  experts  who  are  independent  of  the  program. 


New  SE  Policy  in  Draft  DoDI  5000.02 

Enclosure  12.  Systems  Engineering 

El 2. 5.  Configuration  Management.  The  PM  shall  use  a  configuration 
management  approach  to  establish  and  control  product  attributes  and  the 
technical  baseline  across  the  total  system  life  cycle.  This  approach  shall 
identify,  document,  audit,  and  control  the  functional  and  physical 
characteristics  of  the  system  design;  track  any  changes;  provide  an  audit  trail  of 
program  design  decisions  and  design  modifications;  and  be  integrated  with  the 
SEP  and  technical  planning.  At  completion  of  the  system  level  Critical  Design 
Review,  the  PM  shall  assume  control  of  the  initial  product  baseline  for  all 
Class  1  configuration  changes. 

El 2.6.  Environment,  Safety,  and  Occupational  Health  (ESOH).  The  PM  shall  use 
the  methodology  in  MIL-STD-882D  to  assess  ESOH  risk,  eliminate  ESOH 
hazards  where  possible,  manage  the  risks  that  cannot  be  eliminated,  and  report 
on  the  status  of  ESOH  risk  at  technical  reviews. 

E12.6.1.  Programmatic  ESOH  Evaluation  (PESHE).  The  PM  for  all 
programs,  regardless  of  ACAT  level,  shall  prepare  a  PESHE  and  summarize  it 
in  the  acquisition  strategy. 

El 2. 5. 2.  NEPA/EO  12114.  The  PM  shall  conduct  and  document  NEPA/EO 
12114  analyses,  to  be  approved  by  the  CAE,  for  which  the  PM  is  the  action 
proponent. 

E12.6.3.  Mishap  Investigation  Support.  The  PM  will  support  system- 
related  Class  A  and  B  mishap  investigations. 


New  SE  Policy  in  Draft  DoDI  5000.02 

Enclosure  12.  Systems  Engineering 


El  2.7.  Corrosion  Prevention  and  Control.  Each  AC  AT  I  program  shall  document  its 
strategy  in  a  Corrosion  Prevention  Control  Plan  at  Milestones  B  and  C. 

El 2. 8.  Modular  Open  Systems  Approach  (MOSA).  Program  managers  shall  employ 
MOSA. 

El  2.9.  Data  Management  and  Technical  Data  Rights.  Program  Managers  for  AC  AT 
I  and  II  programs,  regardless  of  planned  sustainment  approach,  shall  assess  the 
long-term  technical  data  needs  of  their  systems  and  reflect  that  assessment  in  a 
Data  Management  Strategy  (DMS). 

E12.10.  Item  Unique  Identification  (TUID).  To  enhance  life-cycle  management  of 
assets  in  systems  acquisition  and  sustainment,  and  to  provide  more  accurate  asset 
valuation,  all  PMs  shall  plan  for  and  implement  IUID  to  identify  and  track 
applicable  major  end  items,  configuration-controlled  items,  and  Government- 
furnished  property.  IUID  planning  and  implementation  shall  be  documented  in  an 
IUID  Implementation  Plan  and  summarized  in  the  program's  Systems  Engineering 
Plan  (Reference  (an)  and  DoD  Directive  8320.03,  Reference  (bv)). 

E12.ll.  Reliability,  Availability,  and  Maintainability  (RAM).  PMs  for  all  programs 
shall  formulate  a  viable  RAM  strategy  that  includes  a  reliability  growth  program 
as  an  integral  part  of  design  and  development.  RAM  shall  be  integrated  within  the 
Systems  Engineering  processes,  documented  in  the  program’s  SEP  and  LCSP,  and 
assessed  during  technical  reviews,  T&E,  and  PSRs. 


Implications  for  Systems  Engineering 


New  Opportunities  for  Enhanced  SE  - 
Starting  Programs  Right 


What’s  relevant:  •  Mandatory  Materiel  Development  Decision 

•  Mandatory  Milestone  A  for  all  “major  weapon  systems” 

•  MS  B  after  system-level  PDR*  and  a  PDR  Report  to  the  MDA* 


MSA 


MS  B 


Strategic  Joint 
Guidance  Concept! 


PDR 


Engineering  and 
Manufacturing 
Development  and 
Demonstration 


T 

CDR 


MS  C 


CPD 

P 


Production  and 
Deployment 


O&S 


Full  Rate  Production 
Decision  Review 


Pre-MDD  “ SE  Touch 
Points " 

•  Initial  Capabilities 
Document  (ICD) 

•  Analysis  of  Alternatives 
study  plan 


Pre-Milestone  A  “ SE  Touch  Points ” 

•  Systems  Engineering  Plan 

•  Technology  Development  Strategy 

•  Test  and  Evaluation  Strategy 

•  Analysis  of  Alternatives 


*  PDR  -  Preliminary  Design  Review 


*  CDR  -  Critical  Design  Review  *  MDA  -  Milestone  Decision  Authority 


SE  Focus:  Materiel  Solution  Analysis 


MDD 


MSA 


SE  Focus:  Technology  Development 
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User  assessment  of  capability  needs 
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Other 

Program 

Activities 


Actions 
Depending 
on  SE  input 


Typically  executed  by  PMO  SE  Staff 

Typically  executed  by  Industry 

^  Delivered  Product 

Mandated  Preliminary  Design  Review 
Technical  Review 
☆  Technical  Assessment 

Grey  Areas  depending  on  PMO  SE  input 

- ►  Leads  to 

- ►  Informs 

- ►  Key  SE  input 


Final 

System  Spec 

SEP 

ISP 
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for  Initial 
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Prototyping  for  CTE  and  for  design  may  be  independent  efforts 
May  vary  with  contracting  strategy  (e.g.,  multiple  designs) 


New  Opportunities  for  Independent  Reviews 


What’s  relevant:  •  Mandatory  Milestone  A  for  all  “major  weapon  systems” 

•  MS  B  after  system-level  PDR*  and  a  PDR  Report  to  the  MDA 

•  EMDD  with  Post-CDR*  Report  and  MDA  Assessment 

•  PSR  and  AOTR  in  policy 


MS  A  MS  B 


MS  C 


Strategic  Joint 

CBA 

Guidance  Concepts 
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Materiel 

Solution  Technology 
Analysis  Development 


Potential  Independent  Technical  ^ 
Reviews  -  PSRs  ^  and  AOTRs  ^ 
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Manufacturing 
Development  and 
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Production  and 
Deployment 
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^  ^  Full  Rate 

I  M  Production 

CDR  OTRR  “ 


Program  Support  Reviews  (PSRs) 

❖  All  ACAT  ID  &  1AM 

❖  To  inform  the  MDA  on  technical  planning  and 
management  processes  thru  risk  identification 
and  mitigation  recommendations 

❖  To  support  OIPT  program  reviews  and  others 
as  requested  by  the  MDA 


Assessments  of  Operational  Test 

Readiness  (AOTRs) 

❖  All  ACAT  ID  and  special  interest  programs 

❖  To  inform  the  MDA,  DOTE,  &  CAE  of  risk  of 
a  system  failing  to  meet  operational  suitability 
and  effectiveness  goals 

❖  To  support  CAE  determination  of  materiel 
readiness  for  IOT&E 


*  PDR  -  Preliminary  Design  Review 


*  CDR  -  Critical  Design  Review 


*  OTRR  -  Operational  Test  Readiness  Review 


Backup 


Draft  DoD  Instruction  5000.02  Extract 


Milestone  A  (per  FY’08  NDAA  Sec.  943) 

“The  project  shall  enter  the  Technology  Development  Phase 
at  Milestone  A  when  the  MDA  has  approved  the  TDS.  The 
tables  in  Enclosure  3  identify  all  statutory  and  regulatory 
requirements  applicable  to  Milestone  A.  . . .  The  MDA  shall 
comply  with  the  certification  requirements  at  Milestone  A  as 
described  in  Enclosure  10  of  this  Instruction.  This  effort 
normally  shall  be  funded  only  for  the  advanced 
development  work.  Technology  development  for  a  major 
weapon  system  shall  not  proceed  without  Milestone  A 
approval.  For  business  area  capabilities,  commercially 
available  solutions  shall  be  preferred.  A  favorable  Milestone 
A  decision  DOES  NOT  mean  that  a  new  acquisition  program 
has  been  initiated.” 


Configuration  Steering  Boards 


Configuration  Steering  Boards  (CSB).  The 
Acquisition  Executive  of  each  DoD  Component 
shall  establish  a  CSB  with  broad  executive 
membership  including  senior  representatives  from 
the  Office  of  the  USD(AT&L)  and  the  Joint  Staff. 

•  The  CSB  shall  review  all  requirements  changes 
and  any  significant  technical  configuration 
changes  for  ACAT  I  and  IA  programs  in 
development  which  have  the  potential  to  result  in 
cost  and  schedule  impacts  to  the  program.  Such 
changes  will  generally  be  rejected,  deferring  them 
to  future  blocks  or  increments.  Changes  shall  not 
be  approved  unless  funds  are  identified  and 
schedule  impacts  mitigated. 

•  Program  Managers  shall,  on  a  roughly  annual 
basis,  identify  and  propose  a  set  of  descoping 
options  to  the  CSB  that  reduce  program  cost  or 
moderate  requirements.  The  CSB  shall 
recommend  to  the  MDA  (if  an  ACAT  ID  or  1AM 
program)  which  of  these  options  should  be 
implemented.  Final  decisions  on  de-scoping 
option  implementation  shall  be  coordinated  with 
the  Joint  Staff  and  military  department 
requirements  officials. 
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Test  and  Evaluation 
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Accelerate  Performance  Improvements: 

Systems  Engineering  Skills  Competency 
Analysis  and  Training  Program  Development 


Steven  A.  Diebold 
Director,  Future  Force  Systems  Engineering 
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Agenda 


•  GDLS  Overview 

•  SE  Training  &  Education  Program  Overview 

•  Competency  Assessment 

•  Gap  Analysis 

•  Curriculum  Development 

•  Results  to  Date 

•  Future  Activities 
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GDLS  Mission 


General  Dynamics  Land  Systems  provides  a 
full  spectrum  of  land  and  amphibious  combat 
systems,  subsystems  and  components 
worldwide 

Our  strengths  are  world-class  design  and 
systems  integration,  superior  production  and 
innovative  life  cycle  support 


We  will  deploy  these  strengths  to  meet  our 
customers’  needs  in  a  changing  world 
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U.S.  Locations 


Ft.  Wainwright 

Ft.  Richardson 

Muskegon 
Technical 
Center 


GDLS 

Central  Office  - 
GDLS  Logistics  & 
Engineering  Center 


Ft  Lewis 


Schofield  Barracks 


Pohakuloa 
Training  Area 


Camp  Pendleton 


Joint  Systems 
Ft.  Hood  Manufacturing  Anniston 

Center  Army  Depot 


GDLS  Future 
Combat  Systems 


Shelby 
Operations 

Scranton 
Operations 

Robotic 
Systems 

Amphibious 

Systems 

Tallahassee 

Operations 
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Major  Contributors  to  Poor  Program 
Performance* 


•  Lack  of  technical  planning  and  oversight 

•  Inadequate  understanding  of  requirements 

•  Incomplete,  obsolete,  inflexible  and  Stovepipe  Physical  and 
Functional  architectures 

•  Stovepipe  developments  with  late  integration 

•  Lack  of  subject  matter  expertise  at  the  integration  level 

•  Low  visibility  of  software  risk 


Lack  of  systems  engineering  discipline, 
authority,  and  resources 


GENERAL  DYNAMICS 

Land  Systems 


*  DoD-directed  Studies/Reviews,  2005 
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GDLS’s  Response 


•  Organize  along  Product  Centers 

i - >  Voice  of  customer 

•  ‘One  Engineering  Design  and  Development  Team’  for 
GDLS 

71  Integrated  Process  System  across  all  Locations 
7i  CMMI  Level  3/5 

•  Revitalize  Systems  Engineering 

71  Process  Improvements 

■  Gate  Reviews,  Six  Sigma,  DFR 
7i  SE  Training  &  Education  Program  Today’s  Topic 
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SE  Training  &  Education  Program 
Development  Overview 


•  Roles  identified  for  Systems  Engineering 

•  For  each  role,  required  competencies  established 

•  Employees  assessed  against  required 
competencies  for  their  assigned  roles 

•  Results  of  competency  assessments  analyzed  to 
identify  gaps 

•  SE  Curriculum  developed  to  address  high  and 
medium  gaps  and  to  further  develop  employees 
with  low  or  no  gaps 

•  Training  Plan  developed  to  incorporate  SE 
Curriculum,  mandatory  courses,  and 
Seminars/Conferences 

•  Progress  to  goals  and  training  effectiveness 
measured  by  Level  1  evaluations 
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Competency  Assessment 


•  Supervisor  verifies  that  correct  roles  are  assigned  to  Employee 

•  Employee  conducts  self-assessment  of  competency  levels  for 
each  required  competency 

Basic  -  Trained  or  understands  basic  concepts  of  the  competency, 
however  still  needs  help  in  applying  the  competency 

Qualified  -  Has  a  good  command  of  the  competency,  no  help  needed 
in  applying  the  competency 

Advanced  -  Has  advanced  understanding  of  the  competency,  can 
lead  and/or  teach  others  in  applying  the  competency 

None  -  Does  not  meet  basic  competency  level 

•  Supervisor  verifies  assessment 

•  Training  Coordinator  compiles  all  completed  assessments 

•  Training  Coordinator  evaluates  roles  to  determine  which  roles 
represent  80%  of  the  Systems  Engineering  population 


GENERAL  DYNAMICS 
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Gap  Analysis  Methodology 


Determined  which  roles  represent  80%  of 
the  SE  population  (top  roles) 

I  I 

Identified  top  20  required  competencies  for 
the  top  roles 

I  I 

Analyzed  results  of  competency  assessments  to 
determine  distribution  of  gaps  across  the  top  roles  for 
the  top  20  required  competencies 
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Competency  Assessment  Results 


•  Highest  Gap 

71  SE  Principles 
71  Project  Management 
71  Domain  Specific  Skills 

•  Medium  Gap 

71  Risk  Analysis 

71  Test  &  Validation  Planning 

71  Baseline  Management  (CM) 

•  Lowest  Gap 

71  Requirements  Management 
71  Trade  Studies 
71  Reliability 
7i  Design  Integration 
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SE  Curriculum 


Low  or  No  Technical  Gaps 

SSCI  SE 
Certificate 
Program 

Certified  SE 
Professional 
(CSEP) 

By  2011: 

10%  Earn  SSCI  SE 

Certificate  (68  total) 

By  2011: 

10%  Earn  INCOSE  CSEP 
(68  total) 

High  &  Medium  Technical  Gaps  | 

SE  Overview/ 
SE  Principles 

Basic 

Configuration 

Management 

By  2011: 

100%  Complete  SE 

Overview/Principles 
(676  total) 

In  2008: 

25  students  complete  Basic 
Configuration  Management 

*  Based  on  676  SE  employees  (Contractors  not  included) 


Knowledge 
Retention  & 
Development 


Design  for  Six 
Sigma 


Design  for 
Reliability 
Curriculum 


Cross 

Functional 

Development 


Risk  Analysis 
Succession  Planning 
Succession/Leadership 
Development 
Conferences  &  Seminars 


Master  Black  Belt  TTT 
DFSS  Green  Belt  Program 


Developed  with  outside 
vendor  (Air  Academy)  to  be 
delivered  in-house  by 
GDLS  Six  Sigma  & 
Emerging  Methods 


Rotational  job  assignments: 
Logistics  Engineer 
LSE 

Section  Manager 


SE  Re-Vitalization  -  Skills  &  Organizational  Feedback 
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Development  of  SE  Training  Plan 
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Training  Budget  Distribution 

•  Training  represents  6%  of  the  Systems  Engineering  overhead  budget. 


•  Mandatory  Training 

includes  health, 
safety  and  security 
courses. 


•  Non  Technical  Training 

includes  courses  such  as 
leadership  development, 
teaming,  CMMI  and  ISO. 


•  Technical  Training 

includes  courses  such 
as  SE  Certificate 
Program/Overviews, 
GD&T,  Soldering  and 
Welding. 


GENERAL  DYNAMICS 

Land  Systems 


Approved  for  Public  Release,  Distribution  Unlimited,  GDLS  Approved,  Log  2008-87, 

Dated  09/30/08 


N-539  1 3 


Development  of  SE  Courses 


2006 

•  Training  Gap  Analysis  of  Systems  Engineering  employees  revealed  need  for  Systems 
Engineering  courses. 

•  Completed  trade  study  and  selected  Center  for  Systems  Management  (CSM)  based 
largely  on  their  affiliation  with  Stanford  University. 

•  Delivered  first  sessions  of  SE  courses  with  CSM. 

2007 

•  CSM/Stanford  University  no  longer  affiliated. 

•  Second  trade  study  conducted  to  determine  if  vendor  change  best  option  for  future 
course  delivery. 

•  Systems  and  Software  Consortium  (SSCI)  selected  based  on  reputation  and  prior 
relationship. 

•  Collaborated  with  SSCI  to  tailor  standard  course  materials  for  GDLS. 

•  Delivered  first  sessions  of  12-day  SE  Certificate  Program  (SECP). 

2008 

•  Continued  offerings  of  SECP  and  added  5-day  and  2-day  SE  Overview  course  to 
training  plan. 

•  Utilized  Michigan  Economic  Development  Grant 


GENERAL.  DYNAMICS 

Land  Systems 


Approved  for  Public  Release,  Distribution  Unlimited,  GDLS  Approved,  Log  2008-87, 

Dated  09/30/08 


N-539  14 


Training  Goals 


100 


"O  75 


50 


CO  25 


SE  Certificate  Program 


□  Projected  ■  Actual 


31 


2007  2008  2009  2010  2011 

Cumulative  Totals 


2007 


2008  2009  2010 

Cumulative  Totals 


2011 
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Training  Evaluation 

Levels  of  Evaluation 


Measures:  Impact  of  training  on  business  performance 

Tools:  Average  Competency  Level  of  students  vs.  Delivery 
Cost  per  student,  process  performance  measures 


Measu res :  The  transfer  of  skills/knowledge  to 
employees’  work 

Tools:  Employee  competency  level  assessed  by 
Supervisor,  Employee  confidence  level  self-assessed 


Measures:  Extent  to  which  students  have 
advanced  skills/knowledge 

Tool:  Pre-  and  Post-tests 


Measures:  Students  reaction  to  the 
training 

Tool:  Surveys 
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Results  to  Date 


•  Development  of  evaluation  methods:  surveys,  pre/post 
testing,  90-day  evaluations 

•  Evaluations  reveal  effectiveness  of  courses 

•  Student  comments  used  to  improve  future  course 
delivery 

•  Modest  changes  to  2006  SE  Training  Gap 


GENERAL  DYNAMICS 

Land  Systems 


Approved  for  Public  Release,  Distribution  Unlimited,  GDLS  Approved,  Log  2008-87, 

Dated  09/30/08 


N-539  1 7 


SE  Training  Effectiveness 

-  Level  1  Evaluation:  Course  Surveys  administered  at  end  of  class 

17  question  survey  used  to  evaluate  students’  satisfaction  with  the  course 
content,  instructor,  resources,  and  relevance  of  course  to  their  jobs. 

EXAMPLE  COURSE  1: 

ANALYSIS 

■  Some  attendees  were  employees 
with  many  years  of  experience  and 
felt  that  the  course  was  not 
relevant  to  them. 

■  Course  material  needs  to  be  made 
more  relevant  to  SE.  Too  much 
focus  on  Software. 


CORRECTIVE  ACTION 

■  One  time  offering.  No  action  to  be 
taken  at  this  time.  If  future 
offerings  to  be  scheduled,  consider 
tailoring  course  material  to  SE  and 
use  updated  gap  analysis  data  to 
identify  attendees. 


EXAMPLE  COURSE  1 


100.0% 


90.0% 


U) 

|  80.0% 
(0 
£ 


70.0% 


60.0% 


%  Rating 


GOAL:  81-100  5  Very  Satisfied 


61-80  4  Somewhat  Satisfied 
41-60  3  Neutral 

21-40  2  Somewhat  Dissatisfied 
00-20  1  Very  Dissatisfied 


Course  Content 


Instructor 


Relevance 


Resources 


Category 


GENERAL  DYNAMICS 

Land  Systems 


Approved  for  Public  Release,  Distribution  Unlimited,  GDLS  Approved,  Log  2008-87, 

Dated  09/30/08 


N-539  1 8 


SE  Training  Effectiveness 


-  Level  2  Evaluation:  Pre/Post 


SE  Overview  5  Day  Pre/Post-Test 
Class  Held:  8-18-22-08  -  VIS  Room 

Employee  #  _  \^\  Pre-Test  \^\  Post-Test 

Please  take  a  few  minutes  to  answer  the  questions  below  to  the  best  of  your  ability.  This  is  a  two-part  exercise  with  the 
purpose  of  measuring  the  basic  skills/knowledge  gained  through  this  course.  You  will  be  asked  to  complete  this  same  test 
at  the  end  of  the  course.  There  is  no  penalty  for  wrons  answers. 


Possible  Answers: 

Maintenance 

Process  Control 

State 

Measures  of  Effectiveness 

Quality 

Systems  Engineering 

Mode 

Reliability 

Validation 

Planning 

SEMP 

Verification 

Fill  in  the  blanks  using  the  choices  above. 

1 .  _ is  an  interdisciplinary  approach  and  means  to  enable  the  realization  of  successful 

systems. 


2.  _ are  used  to  quantify  the  performance  of  system  products  and  processes. 

3.  _ is  the  condition  of  the  system. 

4.  _ is  the  manner  in  which  the  system  operates. 

5.  _ and _ are  elements  of  Logistics  Support. 

6.  _ answers  the  question,  “Did  we  build  the  right  thing?” 

7.  _ describes  engineering  specialty  integration,  the  SE  work  to  be  done,  and  the 

management  of  this  work. 

8.  _ and _ are  engineering  specialties. 


9.  How  would  you  rate  your  current  knowledge  of  Systems 
Engineering?  (Check  one.) 


10.  What  did  you  gain  from  this  course?  (Check 
one.) 


□  Advanced  -  Have  an  advanced  understanding  of  the  skill, 
can  lead  and/or  teach  others  in  applying  it. 


□  New  ideas  related  to  Systems 
Engineering 


□  Qualified  -  Have  a  good  command  of  the  skill,  no  help 
needed  in  applying  it. 


□  Build  on  my  existing  knowledge  of 
Systems  Engineering 


□  Basic  -  Understand  basic  concepts,  but  still  need  help 
applying  it. 


□  Basic  knowledge  of  Systems 
Engineering 


□  None  -  Have  no  knowledge  of  this  skill/topic. 


□  Nothing 


Please  return  this  test  to  the  instructor  when  you  have  completed  it. 


Testing  Jest  of  10  questions  based  on 
course  content  administered  at 
start  and  end  of  courses  to 
measure  initial  effectiveness  of 

course  delivery. 


EXAMPLE  COURSE  2: 

ANALYSIS 

■  Few  students  scored  higher  on  Post  Test 

■  Focus  of  course  did  not  match  pre/post  test 
questions  well. 

CORRECTIVE  ACTION 

■  Prior  to  next  course  offering,  work  with  course 
instructor  to  develop  a  Pre/Post  test  that  is  more 
relevant  to  the  topics  reviewed  during  the  course. 
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SE  Training  Effectiveness 

-  Level  3  Evaluation:  Application  -  Post-Course  Evaluation 


Use  standardized  evaluation 
form  to  collect  data. 

Send  via  email  to  students  60-90 
days  following  course. 

Measures  frequency  of  skill  use, 
value  of  skill  on  job,  self- 
assessed  proficiency  rating, 
barriers  to  use  on  job. 

EXAMPLE  COURSE  3: 

ANALYSIS 

■  Analysis  of  preliminary  data  shows 
course  is  well  received  and  is 
perceived  by  attendees  to  have  value  in 
their  day  to  day  activities.  Most 
students  would  recommend  this  course 
to  coworkers  and  managers. 
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Path  Forward 


•  Complete  follow  up  Training  Gap  Analysis  by  year-end 

•  Renew  focus  on  closing  identified  training  gaps 

•  Continue  to  tailor/modify  course  delivery  based  on 
student  feedback 

•  Continue  to  develop  and  improve  evaluation  methods 
to  assess  improved  business  performance 
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Contact  Information 


Steven  A.  Diebold 

Director,  Future  Force  System  Engineering 
38500  Mound  Road 
Sterling  Heights,  Ml  48310 
(586)  825-5869 

diebold@qdls.com 

Wendell  Mullison 

Manager,  System  Engineering  Process  Tools  and  Architecture 
38500  Mound  Road 
Sterling  Heights,  Ml  48310 
(586)  825-4118 

mullison@qdls.com 

Kimberly  Swierpel 

SE  Measurement  Analysis  &  Process  Architecture 
38500  Mound  Road 
Sterling  Heights,  Ml  48310 
(586)  825-4779 

swierpel@qdls.com 
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Roles  &  Competencies 


ROLES  REPRESENTING  80%  OF  SE  POPULATION 

Requirements  Engineer 

Configuration  Management  (CM)  Engineer 

Specialty  Engineer  -  Embedded  Training  Analyst 

Section  Manager 

Team  Lead 

Administrative  Assistant 

Systems  Analysis  Engineer 

Corrective  Action  Engineer 

Environmental  Test  Technician 

Physical  Architect 

CM  (Configuration  Management)  Analyst 

Process  Engineer 

Specialty  Engineer 

System  Integration  Engineer 

Provisioning  Analyst 

Field  Test  Engineer  -  Vehicle  Test  Engineer 

Environmental  Test  Engineer 

Training  Content  Developer 

System  Architect 

CM  Technician 

Requirements  Management  Analyst 

Logistics  Engineer 

Field  Material  Supply  Specialist 

Field  Test  Engineer  -  Supply  Support  Engineer 

Lead  System  Engineer 

Maintenance  Engineer 

Logistics  Engineering  Liaison 

Reliability  Engineer 

Department  Manager 

System  Safety  Engineer 

Technical  Writer  -  Operations  and  Maintenance  Diagnostics  Engineer  -  Troubleshooting  Developer 

TOP  20  COMPETENCIES 

1 

System  Engineering  Principles 

2 

Job  Specific  Process  knowledge 

3 

Product  knowledge  -  (Tracked,  Wheeled  or  FCS  as  applicable) 

4 

Customer  Satisfaction 

5 

Communication 

6 

Effective  meeting  /  reviews 

7 

EVMS 

8 

Risk  Analysis 

9 

Trade  Studies 

10 

Reliability  theory 

11 

Pro  E 

12 

DOORS 

13 

Requirements  Generation  &  Documentation 

14 

Metric  development 

15 

Program  Management 

16 

Test  &  validation  plan  development 

17 

Cost  estimating  /  proposal  development 

18 

DFMEA  principles  &  techniques 

19 

XFMEA  (reliasoft  suite  of  tools  -  Vmetric,  Weibull,  blocksim) 

20 

Project  Planning 
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SE  Certificate  Program  (SECP) 


•  Is  an  on-site  program  leading  to  a  Systems  Engineering 
Certificate  from  the  Systems  and  Software  Consortium,  Inc. 
(SSCI). 

•  Is  an  intensive,  graduate-level  learning  curriculum  for  experienced, 
practicing  engineers. 

•  Is  a  12  day  program  delivered  in  a  building  block  approach  of  four 
3-day  modules  over  a  two  to  three  month  period  with  self-study, 
classroom,  and  team  project  work. 

•  Is  a  program  that  integrates  INCOSE  SE  Handbook  material  in  an 
effort  to  help  participants  who  are  interested  in  pursuing  the 
INCOSE  Certified  Systems  Engineering  Professional  (CSEP) 
certificate. 

•  Provides  the  ability  to  address  skill/competency  gaps  through 
training 

•  Supports  SE  Revitalization 
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SE  Certificate  Program  (SECP) 


SE  Certificate  Curriculum 
(4  classes,  3  days  each) 


Manage  the  Work 

SE  Management 
Organization  &  Systems 
Engineering 

Risk  &  Opportunity  Management 
Technical  Parameter  Measurement 
Work  Breakdown  Structure 
Earned  Value  Management 
Scheduling 
Process 


Close  the  Loop 

Verification 

Deployment 

Validation 

Logistics  Support 

Integration 

Define  the  Solution 


Architecting  and  Synthesis 

Allocation 

Cost  Factors 


Analysis  and  Decisions 
Specialty  Engineering 
Integrated  Product  Teams 


Define  the  Problem 


SE  Planning 

Needs,  expectations  and  constraints 
Concept  of  Operations 


Requirements 
Functional  Analysis 
Specifications 
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Other  SE  Courses 


SE  Principles 

•  Is  an  on-site  courses  developed  by  the  Systems  and  Software 
Consortium,  Inc.  (SSCI). 

•  Offered  as  2  and  5  day  courses 

•  Provides  overview  of  SE  for  inexperienced  engineers  (high  or 
medium  technical  competency  gap). 

•  Describes  the  basics  of  systems  engineering  -  what  it  is,  how  it 
proceeds  through  the  life  cycle  and  why  it  needs  to  be  done. 

Basic  Configuration  Management 

•  Is  a  two-day,  on-site  course  developed  by  the  Systems  and 
Software  Consortium,  Inc.  (SSCI). 

•  Provides  a  foundation  in  basic  Configuration  Management 
principles  and  skills 
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Certified  SE  Professional 


Certified  Systems  Engineering  Professional  is  a  recognized  certification  that 
confirms  that  an  individual  has  the  basic  skills  to  perform  fundamental  Systems 
Engineering  tasks  and  is  able  to  make  a  productive  contribution  to  work  efforts. 

Benefits  of  CSEP  Certification 

•  Formally  recognizes  SE  capabilities  •  Provides  a  competitive  advantage 

•  Distinguishes  CSEP  holder  from  •  Furthers  professional  SE  development 


others  within  a  professional  field  •  Helps  advance  the  art  and  practice  of  SE 

Certification  Process  _ 
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Systems  Engineering  in  DoD 


Nicholas  Torelli 

Systems  &  Software  Engineering 
Office  of  the  Deputy  Under  Secretary  of  Defense 
for  Acquisition  and  Technology 


21  October  2008 


The  Problem  and  Root  Causes 


Systems  and  Software  Engineering 


•  Problem  Statement:  Defense  Acquisition  programs  are  experiencing 
significant  problems: 

-  over  cost 

-  behind  schedule 

-  not  operationally  suitable  or  effective 

•  Root  Causes: 

-  The  Defense  Acquisition  workforce  has  experienced  significant  "peace 
dividend"  and  "baby  boomer"  losses  in  critical  personnel 

-  I  mplementation  of  Acquisition  Reform  went  too  far  in  terms  of 
streamlining  or  reducing  policies  and  processes 

•  The  Department  lacks  adequately  defined  and  enforceable  criteria  to  assess 
program  maturity  at  milestones  with  direct  linkage  to  technical  reviews 

-  Incomplete,  ineffective  and/or  unrealistic  acquisition  strategies  and 
plans  have  resulted  in  poor  program  performance 

-  Poor  or  incomplete  Requirements  development  process 
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Proposed  Solutions 


Systems  and  Software  Engineering 


•  Solutions: 


-  Early  /  Enhanced  Life  Cycle  Engagement  in  Systems 
Engineering 

-  Human  Capital  Strategic  Plan 

-  Systems  Engineering  Research 
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Enhanced  Systems  Engineering 


Systems  and  Software  Engineering 


•  Actions: 

-  Fostered  Enhanced  Systems  Engineering  Policy  in  DoDI  5000.02 

•  Refined  SE  content  through  out  the  Acquisition  Life  Cycle 
(Milestones  /  Mandatory  Technical  Reviews) 

•  Detailed  SE  uniquely,  in  DoDI  5000.02  -  Enclosure  12 

-  Established  new  policy  on  key  SE  Design  Considerations 
(Reliability,  Availability,  Maintainability  (RAM)) 

-  Promulgated  focused  and  expanded  SE  Guidance  IAW  Policy 

•  Formalized  design  reviews  and  SE  Processes  for  accountability 

•  Authored  sections  of  Defense  Acquisition  Guidebook  update 

•  Partnered  in  establishing  RAM-C  Guidebook  and  Contract  Language 

•  Continuing  updates  to  Defense  Acquisition  Program  Support 
methodology  supporting  Program  Support  Reviews 


“Implement  the  right  activities  at  the  right  time  in  the  right  way ” 
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Human  Capital  Strategic  Plan 


Systems  and  Software  Engineering 


•  Actions: 

-  Improving  the  Defense  Acquisition  Workforce  by: 

•  Recruiting  and  Hiring  Qualified  Personnel  /  Highly  Qualified 
Experts 

•  Training  and  Developing  Defense  Acquisition  Personnel 

•  Retaining  and  Recognizing  Qualified  Personnel 

-  Evaluating  and  Improving  SE  Competencies  through: 

•  Education  (Universities  and  associated  Service  Colleges) 

•  Training  (DAU) 

•  Experience  Opportunities  (e.g.,  rotations,  OJT) 
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Systems  Engineering  Research 


Systems  and  Software  Engineering 


•  Actions: 

-  Systems  Engineering  Research 

•  Established  SE  Research  University  Affiliated  Research  Center 
(UARC)  at  Stevens  Institute  of  Technology 

-  Technical  Task  Order-based  research  opportunities 
»  OSD  /  Components  fund  desired  research 
»  Knowledge  shared  across  all  associated  universities 
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Systems  Engineering  Knowledge  Map 


Knowledge 

Development 

Systems  and  Software  Engineering 

Combining  existing 
knowledge  from  different 
sources  into  new  and 
useful  working  knowledge 


Distributing  knowledge 
from  those  who  have  it  to 
those  who  need  it 


Discovering  new 
knowledge  from  research, 
program  assessments, 
industry  best  practices 
and  other  sources 


Knowledge 

Distribution 


Knowledge 

Discovery 
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Systems  Engineering  Knowledge  Map 


Systems  Engineering 
Functions 


Guidance 
and  Tools 


Knowledge 

Distribution 


Knowledge 

Development 


Systems  and  Software  Engineering 


Enabling 

Activities 


1  Policy 

Program 

Assessments 

Systems  Engineering 
Competencies 

/ 

Professional  Societies 
(INCOSE,  NDIA,  etc.) 


IX 


Best  Practices 


Industry 


\ 

|  Outreach  | 

/ 

, - \ 

Research  (UARC) 

1  Training  1  \ 

> 

Education  I 


Academia 


Knowledge 

Discovery 
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Systems  Engineering  Knowledge  Flow 


Perform 


Acquisition 

Programs 


Workforce 
Capabilities  & 
Demographics 


Individual 

/Assessment 


Individual 

Development 

Plans 


Assessment 


Human 

Capital 

Strategic 

Plan 


Change 


Adjust 


Cultural 

Barriers 


Workforce 

Shortfalls 


Workforce 
Assessment 
Results 


Analysis 


[Request 
\ Training 


Revise 

Curriculum 


Training 

Shortfalls 


Provide 
I  Training 


AT&L  Performance 
Learning  Model 
(DAU) 


Revise 
Competency 
Model 


Competency 

Gaps 


Develop  /  Update 
Curriculum 


Competency 

Model 


Recruit 

Train 
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INCOSE,  NDIA,  etc. 
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Research 


Program  Support  Reviews 
(PSRs)  and  Assessments 


Assessments 

Results 
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Program 
Managers 


Program 

Performance 


Policy  or 
Guidance 
Shortfalls 


Revise 


Policy 

and 

Guidance 


UARC 


Develop  / 
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Policy, 
Guidance  & 
Competency 
Model 


Systems  Engineering  Policy 


Systems  and  Software  Engineering 


•  Draft  OSD  Acquisition  Policy  (DoDI  5000.02)  is  in  for  final  signature  ... 
substantial  changes  to  the  early  acquisition  process  (in  consonance  with 
NRC  Study),  including 

-  Mandatory  Materiel  Development  Decision  (MDD) 

-  Mandatory  competing  prototypes  before  MS  B 

-  Mandatory  PDR  and  report  to  the  MDA  before  MS  B 

-  Configuration  Steering  Boards  at  Component  level  to  review  all  requirements 
changes 

-  Mandatory  government  control  of  Class  I  changes  no  later  than  CDR  for 
Configuration  Management 

•  Renewed  emphasis  on  manufacturing  during  system  development: 

-  Re-titles  SDD  phase  to  EMDD  with  two  sub  phases:  Integrated  System  Design 
and  System  Capability  and  Manufacturing  Process  Demonstration 

-  Establishes  consideration  of  manufacturing  maturity  at  key  decision  points 

•  Mandatory  system-level  CDR  with  an  initial  product  baseline  followed  by  a 
Post-CDR  Report  to  the  MDA 

•  Post-CDR  Assessment  by  the  MDA  between  EMDD  sub-phases 

This  includes  explicit  recognition  of  Systems  Engineering  in  all  phases,  but 
especially  early  in  the  acquisition  life  cycle 
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Systems  Engineering  Guidance 


Systems  and  Software  Engineering 


•  Plans  are  underway  to  complete  the  update  of  all  Systems 
Engineering  (SE)  documentation  based  on  the  updated  Policy: 

-  Defense  Acquisition  Guidance  (DAG)  Chapter  4  (SE) 

-  Systems  Engineering  Plan  (SEP) 

-  Integration  of  Systems  Engineering  into  Contracts 

-  Defense  Acquisition  Program  Support  (DAPS)  methodology 

•  I  mpacting  Requirements  Generation  earlier  through  J  oint  Staff 
recommendation  for  Capability  Description  Document  early  in  the 
Technology  Development  phase  to  influence  system  design 

•  Published  System  of  Systems  Guide,  Modeling  &  Simulation  Guide, 
Test  &  Evaluation  Contracts  Guide 

•  Tools 

-  Acquisition  Guidance  Model 
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Human  Capital  Initiatives 
(SE  Education  and  Training) 


Systems  and  Software  Engineering 


•  Re-coding  of  program  level  engineering  specialty  positions  to  Program 
Systems  Engineer  (PSE)  is  in  progress  across  the  Services. 

-  Added  additional  training  and  experience  requirements 

•  Focus  on  enhancing  SE  in  the  early  phases  of  acquisition 

•  Broaden  the  competency  set  to  include  other  career  fields  (e.g.,  PM,  Logistics, 
Contracting) 

•  Double  the  years  of  experience  required  for  each  DAWIA  certification  level 

•  Conducting  Systems  Engineering  Competency  Assessment  in  late  2008 
/  early  2009  (based  on  SME  validation  of  competency  model,  to  be 
completed  in  November  2008) 

•  Key  contributors  to  DAU's  "Requirements  Manager"  training  curriculum 
for  Joint  Staff  /  Services  personnel  who  develop  and  manage 
requirements 

•  Surveying  SE  Education  curricula  and  programs  for  future  leverage 
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Human  Capital  Initiatives 
(SE  Education  and  Training) 


Systems  and  Software  Engineering 


Defense  Acquisition  Workforce  Development  Fund  (based  on  NDAA 
Section  852,  Defense  Acquisition  Workforce  Development  Act) 

-  Recruiting  and  Hiring: 

•  I  ntern  Programs. 

•  Recruiting  Incentives. 

•  Outreach  Programs. 

•  Journeyman  Hiring  Programs. 

•  Hiring  Expert  Knowledge  -  Highly  Qualified  Experts  (HQE). 

-  Training  and  Development: 

•  Training  Enhancement  and  Capacity  Expansion. 

•  Comprehensive  Acquisition  Workforce  and  Student  Information  System. 

•  Competency  Management  and  Assessments. 

•  Workforce  Planning  Pilot  Program. 

-  Retention  and  Recognition: 

•  Retention  and  Recognition  Incentives. 

•  Career  Broadening  and  Academic  Programs. 
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Human  Capital  Initiatives 

(Defense  Acquisition  Workforce  Development  Fund 1) 


Systems  and  Software  Engineering 


1  Based  on  NDAA  Section  852,  Defense  Acquisition  Workforce  Development  Act 
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Examples  of  SSE  Outreach  (1)  33C 

"  ^  ~  Systems  and  Software  Engineering 


•  Conducted  cross-Service  /  OSD  PDR  Workshop,  examining  the 
impact  of  the  movement  of  PDR  prior  to  Milestone  B  decision  point. 

-  Developed  updates  /  improvements  to  the  draft  Guidance  based  on  the 
results 

•  Defense  Acquisition  Program  Support  (DAPS)  methodology  used  by 
SSE  for  Program  Support  Reviews  is  being  shared  with  the  Services 

•  Best  Practices  Clearinghouse  -  focused  effort  to  leverage  this 
Defense  Acquisition  University  asset  to  provide  an  accessible 
repository  of  lessons  learned  and  best  practices  across  DoD  and 
other  agencies  (e.g.,  NASA) 

•  Co-Chair  of  NDIA  SE  Division  Education  and  Training  Committee 
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Examples  ofSSE  Outreach  (2)  33C 

"  ^  ~  Systems  and  Software  Engineering 


•  Assisted  INCOSE  (International  Council  on  Systems  Engineering)  in 
development  of  a  certification  program  for  Systems  Engineers  who 
work  on  DoD  Acquisition  programs,  based  directly  on  the  Defense 
Acquisition  Guidance  (DAG).  The  designation  is  "CSEP  -  Acq" 

-  Approval  for  DAU  SYS-101  and  -202  equivalency  in  work 

•  Working  with  Naval  Postgraduate  School  SE  Department  and  Air 
Force  Institute  of  Technology  /  Center  for  Systems  Engineering  to 
help  align  their  SE  curriculum  with  Service  and  OSD  policy  and  to 
facilitate  equivalency  with  similar  DAU  SE  courses 

•  Lead  for  2009  Singapore-US  Exchange  Forum  on  Systems 
Engineering;  focus  will  be  on  international  SE  competencies 
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Implementing  the  2007 
Developmental  Test  &  Evaluation 
Defense  Science  Board  Results 

Oct  2008  NDIA  SE  Conference 


Mr.  Chris  DiPetto 

Deputy  Director 


Developmental  Test  &  Evaluation 
OUSD(AT&L)/Systems  &  Software  Engineering 


Problem  Definition 


DEVELOPMENTAL  TEST  &  EVALUATION 


•  Approximately  50%  of  programs  completing  Initial 
Operational  Test  and  Evaluation  (IOT&E)  have  not  been 
evaluated  as  operationally  effective  and  operationally 
suitable.  These  results  in  IOT&E  suggest  deficiencies  in 
our  DT&E  processes. 

>  Substantial  increase  in  the  number  of  systems  not  suitable 
during  IOT&E 

>  Suitability  failures  are  as  high  as  80%  for  some  commodities 

>  Reliability,  Availability  and  Maintainability  (RAM)  deficiencies 
comprise  the  primary  shortfall  areas 


T&E  -  From  Concept  to  Combat 
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Tasking:  Terms  of  Reference 


DEVELOPMENTAL  TEST  &  EVALUATION 

Review,  assess  and  recommend  changes  to  improve: 

>  OSD  T&E  organization,  roles,  and  responsibilities 

>  DT&E  oversight  and  facilitate  integrated  T&E 

>  DT&E  Title  10  authority 

>  DT&E  process  improvements  to  discover  suitability  problems 
earlier 

Additional  Task  Force  Objectives: 

>  Conduct  root  cause  analysis  of  suitability  problems 

>  Recommend  changes  to  correct  systemic  problems 
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Summary  of  Major  DSB  Findings 


DEVELOPMENTAL  TEST  &  EVALUATION 


•  RAM  shortfalls  are  identified  during  DT,  but  program  constraints 
(schedule  and  funding)  often  preclude  incorporating  fixes  and 
delaying  IOT&E 

>  Recent  studies  have  reconfirmed  that  improving  RAM  lowers  Life  Cycle 
Costs  (LCCs) 

•  Service  acquisition  programs  are  incorporating  Integrated  Testing  to 
a  limited  degree  through  varying  approaches 

>  Additional  emphasis  on  Integrated  Testing  will  result  in  greater  T&E 
process  efficiency  and  program  cost  reductions 

•  Large  government  acquisition  personnel  reductions  combined 
with  industry/government  retirements  have  had  a  severe  adverse 
impact  on  acquisition  program  support 


DEVELOPMENTAL  TEST  &  EVALUATION 


Selected  Findings  and 
Recommendations 
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RAM  Findings 


DEVELOPMENTAL  TEST  &  EVALUATION 


•  Acquisition  Reform  implementation  detrimental  to  RAM 

>  With  some  exceptions,  reliability  growth  discontinued  during  SDD  and 
deferred  until  production 

>  Relevant  military  specs  and  standards  cancelled  and  not,  in  all  cases, 
replaced  with  industry  standards 

>  Gvmt  Technical/managerial  workforce  reduced  in  most  PMs  and  test 
organizations 

•  RAM  shortfalls  are  frequently  identified  during  DT 

>  Program  constraints  (schedule  and  funding)  often  preclude 
incorporating  fixes  and  delaying  IOT&E 

•  Examples  of  programs  with  such  serious  RAM  concerns  that  they 
were  precluded  from  proceeding  to  production  until  the  problems 
could  be  corrected. 
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RAM  Recommendations 


DEVELOPMENTAL  TEST  &  EVALUATION 

The  single  most  important  step  necessary  to  correct  high  suitability  failure 
rates  is  to  ensure  programs  are  formulated  to  execute  a  viable  systems 
engineering  strategy  from  the  beginning,  including  a  robust  RAM  program,  as 
an  integral  part  of  design  and  development.  No  amount  of  testing  will 
compensate  for  deficiencies  in  RAM  program  formulation. 

To  this  end,  the  following  RAM-related  actions  are  required  as  a  minimum: 

•  Develop  a  military  standard  for  consistent  RAM  development  and  testing  that 
can  be  readily  referenced  in  future  DoD  contracts 

•  Identify  and  define  RAM  requirements  in  JCIDS  and  incorporate  into  RFP 

•  Make  RAM,  to  include  a  robust  reliability  growth  program,  a  mandatory 
contractual  requirement  and  document  progress  as  a  part  of  every  major 
program  review 

^  Flow-down  RAM  requirements  to  subcontractors 

•  Ensure  an  adequate  cadre  of  experienced  RAM  personnel  are  part  of  the 
Service  acquisition  and  engineering  office  staffs 
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Integrated  Test  and  Evaluation 

Findings 

DEVELOPMENTAL  TEST  &  EVALUATION 

•  Service  acquisition  programs  are  incorporating  integrated  testing  to 
a  limited  degree  through  varying  approaches 

>  Army  has  integrated  DT  and  OT  organizations  into  one  command 

>  Navy  utilizes  a  full-spectrum  RDT&E  approach  to  conducting  Test  & 
Evaluation 

>  Air  Force  employs  Combined  Test  Force  concept  which  consolidates 
test  execution 

•  Additional  emphasis  on  integrated  testing  can  result  in  greater  T&E 
process  efficiency  and  program  cost  reductions 
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Integrated  Test  and  Evaluation 
Recommendations 

DEVELOPMENTAL  TEST  &  EVALUATION 

•  Mandate  integrated  DT  and  OT  planning  and  execution  throughout  the 
program 

>  Require  sharing  and  access  to  all  appropriate  system-level  and  selected 
component-level  test  and  model  data  by  government  DT  and  OT  organizations 
as  well  as  the  prime  contractor,  where  appropriate 

>  Incorporate  data  access  requirements  in  contract 

>  Integrate  test  events,  where  practical,  to  satisfy  OT  and  DT  requirements 

>  Define  which  testing  will  be  accomplished  by  the  prime  contractor,  government 
DT  lead,  and  OT  as  the  lead  agency  prior  to  award  of  contract 

>  Require  an  operational  evaluation  framework  as  a  part  of  the  Milestone  B  TEMP 

•  Make  available  a  cadre  of  operational  personnel  to  support  DT  for  ACAT  I 
and  special  interest  programs,  as  a  minimum 

•  Better  integrate  OTAs  into  the  DR  process  to  include  participation  on  Joint 
Reliability  Maintainability  Evaluation  Team  (JRMET)  or  Corrective  Action 
Review  Board  throughout  DT 
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Implementing  Actions 
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RIWG  Chartered  Feb  2008 


DEVELOPMENTAL  TEST  &  EVALUATION 

•  DUSD(A&T)  and  DOT&E  February  memo  established  working  group 
to  implement  recommendations  to  improve  RAM 

•  Specific  Tasks: 

>  Ensure  execution  of  a  viable  SE  strategy  as  an 
integral  part  of  design  and  development 

>  Ensure  government  orgs  reconstitute  cadre  of 
experienced  T&E  and  RAM  personnel 

>  Integrated  DT  and  OT 

>  ensure  data  access 

>  conduct  T&E  in  an  operationally 
representative  environment  as  early  as 
possible 

•  Report  issued  September  5,  2008 
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RIWG  Accomplishments/Recommendations 

DEVELOPMENTAL  TEST  &  EVALUATION 

1 .  SE  strategy  as  an  integral  part  of  design  and  development 

>  Developed  contract  reliability  guidance;  RFP  language 

S  Based  on  GEIA-STD-0009  -  On  DAU  ACC  website 

>  Drafted  RAM  planning  template 
S  RIWG  Report,  Appendix  1.3.2 

>  Updated  Reliability  scorecard  in  DAG 
S  On  DAU  ACC  website 

>  AT&L  RAM  Policy  Memo  (July  21 , 2008) 


T&E  -  From  Concept  to  Combat 


12 


AT&L  Reliability  Policy  Memo  -  July  2008 


•DEVELOPMENTAL  TEST  &  EVALUATION 


•  Services  directed  to  establish  a 
reliability  improvement  acquisition  policy 

-  Report  back  to  AT&L  w/in  30  days  w/ 
plan  to  implement  policies 

•  Effective  immediately,  it  is  DoD  policy 
for  programs  to  execute  a  RAM  strategy 
that  includes  a  reliability  growth  program 
as  an  integral  part  of  design  and 
development 

-  RAM  shall  be  integrated  w/in  SE, 
documented  in  SEP  and  Life  Cycle 
Sustainment  Plan 

-  Assessed  during  technical  reviews, 
T&E,  and  Program  Support  Reviews 


•USD(AT&L)  Memo,  Jul  21,  2008 
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DEVELOPMENTAL  TEST  &  EVALUATION 


2.  Reconstitute  cadre  of  experienced  T&E  and  RAM  personnel 

S  Provided  DAU  course  material  recommendations  for  RAM 
and  T&E 

S  Recommendations  provided  to  DAU  O-FIPT 
S  Curriculum/certification  recommendations  under  review  by 
each  FIPT 

S  Also  addressing  courses  for  Requirements  Officers 

S  OSD/AT&L  initiative  to  recruit  RAM  and  T&E  expertise 
S  NDAA  SECT  852  Workforce  Development  fund 
S  Considering  competency  alignment  as  an  alternative  to 
Centers  of  Excellence 
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DEVELOPMENTAL  TEST  &  EVALUATION 


3.  Implement  mandated  Integrated  DT  and  OT 

S  Published  DoD-common  Integrated  Testing  definition 
S  Revised  TEMP  format  -  In  DAG  update 
S  Guide  on  Incorporating  T&E  in  Acquisition  Contract 
S  Approval  coordination  for  publication  in  process 
S  Located  at:  http://www.acq.osd.mil/sse/dte/docs 
S  Updated  DAG  with  integrated  test  implementation  guidance 


T&E  -  From  Concept  to  Combat 


15 


Integrated  Test  Implementation 


DEVELOPMENTAL  TEST  &  EVALUATION 


Impediments  To  Full  Implementation: 

•  Common  Understanding  -  Definition 

•  Lack  Of  Guidance  -  Updating  DAG  and  TEMP  Content 

•  Culture  Change  -  Leadership  Needs  To  Engage 


Definition  Signed  By  DUSD(A&T)  And  DOT&E 

Coordinated  Across  Components  and  Services 

“Integrated  testing  is  the  collaborative  planning  and 
collaborative  execution  of  test  phases  and  events  to  provide 

shared  data  in  support  of  independent  analysis,  evaluation 
and  reporting  by  all  stakeholders  particularly  the 
developmental  (both  contractor  and  government)  and 
operational  test  and  evaluation  communities.” 
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%J  Revised  TEMP  Concept 


DEVELOPMENTAL  TEST  &  EVALUATION 

Part  I  Part  II  Part  III  Part  IV 

Introduction  Mgmt  &  Sched  T&E  Strategy  Resources 


Brief  mission  description 

Describe  T&E 

The  philosophy  recognizes 

Include  in  para  form  or  table 

paragraph 

management 

a  T&E  continuum  & 
emphasizes  evaluations 

•Test  articles  needed/event 

System  description 

Common  Data 

Evaluation  Framework  ties 

•Special  equip/  instr  costs 

Brief  Threat  Assessment 

Deficiency  Reporting 

T&E  knowledge  to 

•Target  /  expendable  costs 

Program  Background 

TEMP  Updates 

decisions,  requirements,  etc 

Developmental 

•Threat  representation  costs 

•Manpower  needs 

Key  Capabilities 

Overarching  integrated 
schedule  that  includes 
sequencing 
of  T&E  activities 
(CT,  DT,  OT,  LFT,  M&S) 

Live  Fire 

IOT&E  Readiness  Cert 

Operational 

Certifications 

Reliability  Growth 

Future  Testing 

•M&S  costs 

Linkage  of  decisions  to  evaluations,  requirements,  test  phases,  and  resources 

What 

Who,  When 

Why,  How 

Resources  required 

Include  Joint  requirements  throughout 
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T&E  in  DoDI  5000.02 


DEVELOPMENTAL  TEST  &  EVALUATION 


•  Integrated  Testing 

>  IOT&E  still  separate 

•  Assessment  of  Operational  Test  Readiness 

>  Independent  DUSD(A&T)  assessment  -  informs  OTRR 

•  Capability  Comparison 

>  Additional  perspective  for  programmatic  decisions 

•  Data  Sharing 

>  Goal  is  common  data  set  (contractor,  government)  for  evals 

>  Establishing  &  maintaining  data  “pedigree”  is  key 

•  TES/TDS  at  MS-A 

>  Tailor  content  to  competitive  prototyping  and  preps  for  PDR  (now  prior 
to  MS  B) 

>  Focus  on  TDS  &  ICD 
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Summary 


DEVELOPMENTAL  TEST  &  EVALUATION 

•  2007  DT&E  DSB 

>  Results  published  June  2008 

>  Beginning  to  address  the  systemic  issues  with  DT&E 

•  RIWG  Progress  Update 

>  Report  available 

>  Follow-up  in  December 

•  T&E  in  5000.02 

>  In  final  SD  106  review  for  approval 

>  Publication  expected  Fall  2008 
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Questions  and  Discussion 


DSB  Report:  http://www.acq.osd.mil/dsb/reports.htm 
OSD/DT&E:  http://www.acq.osd.mil/sse/dte/index.html 
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T&E  Metrics  for  Acquisition 
Phases  &  Decisions 


Developmental  Test  &  Evaluation 
OUSD(AT&L)/Systems  &  Software  Engineering 


Purpose 

DEVELOPMENTAL  TEST  &  EVALUATION 

•  Define  T&E  metrics  for  decision  points  and  phases  across  the 
acquisition  life  cycle 

-  Define  appropriate  T&E  execution  and  reporting  measures 

-  Standardize  metrics  to  assess  progress  in  T&E  planning 
and  execution 

-  Convey  value-added  role  of  T&E 
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*  The  purpose  of  T&E  is  to  develop  and  deliver  knowledge 

-  Knowledge  =  actionable  information 

*  T&E  developed  knowledge  informs  decisions  to  reduce  risk  in 
requiring,  acquiring,  and  employing  systems  /  capabilities 

*  T&E  knowledge  is  used  to: 

-  Assess  system  capabilities  /  limitations 

-  Assess  program  progress 

-  Assess  technical  progress 

-  Improve  the  product  and  processes 
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Attributes  Measured 


•  The  metrics  required  are  related  to: 

-  Resources  ($,  people,  ranges,  test  assets) 

-  Errors  /  Problems  (#,  discovery  /  correction  rates,  criticality) 

-  Process  characteristics  (uniqueness,  complexity) 

-  Project  Characteristics  (size,  complexity,  schedule) 

-  Project  Dynamics  (Reqt  chg,  Sched  chg,  Resource  chg) 
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Sample  Integrated  Schedule 


DEVELOPMENTAL  TEST  &  EVALUATION 
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Acquisition  Life  Cycle  and  Phases 

Material  Solution  Analysis 

DEVELOPMENTAL  TEST  &  EVALUATION 


Mi 

Z 

A 

Z 

2T 

M^C 

IOC  ★  FOC 

Fvlate ri e I  Solution 
Analysis 

Tectinology 

Development 

Engineering  &  M^rmfactriring 
Development  &  Demonstration 

FR*><> 

PSD 

OSS 

XV  XV 

XV 

XV 

ITR 

A«R 

tr; 

BR 

9RR9FR 

P^R 

TRA 

CDR  BVR 

TRR 

^RCA 

OTRR 

l«R 

•  Focus:  Assess  potential  materiel  solutions 

•  Decision  Points:  MDD,  ITR,  ASR,  TRA,  MS-A 

•  T&E  Activity: 

-  Review  AoA  for  evaluatability,  identify  discriminators 

-  M&S  to  evaluate  alternatives,  sensitivity  analyses 

•  T&E  Products:  TES 

•  Measures  /  Metrics: 

-  T&E  Strategy  defined 

-  CARD  input 
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Acquisition  Life  Cycle  and  Phases 

Technology  Development 


DEVELOPMENTAL  TEST  &  EVALUATION 


Mi 

z 

A  MS 

z  z 

B  ^  ★  IOC  ★  FOC 

Mate  ri  e  1  Solution 
Analysis 

Technology 

Development 

Engineering  &  Manufacturing  FRP/%  [ 

Development  &  Demonstration  | 

ITR  A«R 

TJU 

BR  9RR9FR  FgR 

TRA 

CDR  BVR  TRR  IV  R 

OTRR 

•  Focus:  Reduce  technology  risk,  determine  technologies  for  system 
integration 

•  Decision  Points:  IBR,  SFR,  SRR,  TRA,  PDR,  MS-B,  EMDD  RFP 

•  T&E  Activity: 

-  Risk  identification  &  investigation 

-  Technology  maturation,  integration,  &  demonstration  in  relevant 
environment 
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Acquisition  Life  Cycle  and  Phases 


Technology  Development  cont. 


DEVELOPMENTAL  TEST  &  EVALUATION 


•  T&E  Products:  Technology  evaluation,  TEMP,  CARD  update 

•  Measures  /  Metrics: 

-  T&E  WIPT  charter  status 

-  TEMP  status  (KPP/KSAs  incorporated,  design  risks,  resources) 

-  M&S,  SIL  capabilities  relative  to  desired  level 

-  Test  point  burn-down  (M&S,  SIL) 

-  Test  time  vs  schedule  (M&S,  SIL) 

-  TRLs  achieved 

-  Risk  mitigation  (Initial  &  current  risk  level) 


T&E  -  From  Concept  to  Combat 


Acquisition  Life  Cycle  and  Phases 

Engineering  &  Manufacturing  Dev  &  Demo 

DEVELOPMENTAL  TEST  &  EVALUATION _ 


MS  A  M! 

zx  Z 

"  2 

^  IOC  ★  FOC 

^T«y»v.lVteteriel  Solution  Technology 

s^h^.  Analysis  Development 

Ennlneering  &  Manufacturing 
Development  Demonstration 

FRP0  F&D  OaS  | 

ITR  AWR^IBR  9RR9FR 

TRA  TRJ 

CDR  gVR  TRR 

aPCA  I9R 

OTRR 

•  Focus:  Develop  a  system  or  increment  of  capability,  reduce 
manufacturing  risk,  &  ensure  supportability.  Also  demonstrate 
system  integration,  interoperability,  safety,  &  utility 

•  Decision  Points:  IBR,  CDR,  SVR,  TRR,  FCA,  MS-C 

•  T&E  Activity: 

-  Risk  reduction  -  System,  manufacturing 

-  Assess  design  maturity 

-  Determine  system  capability  &  limitations 

-  Demonstrate  spec  performance 

-  Estimate  reliability 

-  Assess  information  assurance 

-  Ensure  supportability 
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Acquisition  Life  Cycle  and  Phases 

Engineering  &  Manufacturing  Dev  &  Demo  com 

DEVELOPMENTAL  TEST  &  EVALUATION 

•  T&E  Products:  Developmental  evaluation  reports,  OA,  TEMP 

•  Measures  /  Metrics: 

-  DR  quantity  vs  time  (M&S,  SIL,  HITL,  OAR,  manufacturing) 

-  DR  rate  of  discovery/correction  (design  &  manufacturing) 

-  Test  point  burn-down  (M&S,  SIL,  HITL,  OAR) 

-  Test  time  vs  schedule  (M&S,  SIL,  HITL,  OAR) 

-  Configuration  status  (M&S,  SIL,  HITL,  OAR) 

-  CTP  results  vs  thresholds 

-  CTP  results  vs  time 

-  System  capabilities  (mission  context)  characterized 

-  System  Certifications  (Interoperability,  IA,  Safety) 

-  TRLs 
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Number  of  T est  Points 


Metric  Examples 


DEVELOPMENTAL  TEST  &  EVALUATION 


Date 
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Acquisition  Life  Cycle  and  Phases 

Production  &  Deployment 

DEVELOPMENTAL  TEST  &  EVALUATION  _ 


Ml 

z 

C 

X 

IOC 

★  FOC 

Materiel  Solution 
Analysis 

Technology 

Development 

Ennineering  &  Manufacturing 
Development  &  Demonstration 

PSD 

OSS 

1 

3 

1 

ZX 

zx zx 

zx- zx  zx 

ZX  ZX 

ZX 

zx 

ZX 

ITR 

A«R  IBR 

9RR9FR  P^R 

CDR  BVR 

TRF 

^RCA 

9R 

TRA 

TRA 

OTRR 

•  Focus:  Achieve  an  operational  capability 

•  Decision  Points:  PCA,  OTRR,  PRR,  FRP,  IOC 

•  T&E  Activity: 

-  Operational  effectiveness  &  suitability 

-  Vulnerability  /  Lethality 

-  Production  acceptance  &  Manufacturing  process  control 

-  Deficiency  correction 

-  Reliability 
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Acquisition  Life  Cycle  and  Phases 

Production  &  Deployment  cont. 

DEVELOPMENTAL  TEST  &  EVALUATION 

•  T&E  Products:  Developmental  evaluation  report,  AOTR,  IOT&E 
report  (BLRIP),  LFT&E  report,  TEMP 

•  Measures  /  Metrics: 

-  DR  rate  of  discovery/correction  (design  &  manufacturing) 

-  Test  point  burn-down  (OAR) 

-  Test  time  vs  schedule  (OAR) 

-  TOV&V  (O-level,  l-level,  D-level) 

-  System  Certifications  (Interoperability,  IA,  Safety) 

-  MRLs 

-  Configuration  status  (M&S,  OAR,  Trainers) 

-  Operational  Effectiveness  &  Operational  Suitability 

-  Survivability,  Vulnerability,  &  Lethality 

-  System  capabilities  (mission  context)  characterized 
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Acquisition  Life  Cycle  and  Phases 

Operations  &  Support 

DEVELOPMENTAL  TEST  &  EVALUATION  _ 


2T 

M^C 

IOC 

★  FOC 

Mate  ri  e  1  Solution 
Analysis 

Technology 

Development 

Engineering  &  Manufacturing 
Development  &  Demonstration 

PSD 

O^S 

zx zx 

zx  zx 

zx 

XV  XV 

XV 

XV 

ITR 

A«R  IBR 

TRA 

9RR9FR 

P^R 

TRA 

CDR  BVR 

TRR 

z^PCA 

OTRR 

KIR 

•  Focus:  Sustain  the  system 

•  Decision  Points:  ISR,  FOC 

•  T&E  Activity: 

-  Assess  availability,  reliability,  maintainability 

-  Identification  of  new  capabilities,  improved  supportability 

•  T&E  Products:  Deficiency  Reports,  TTP  updates 

•  Measures  /  Metrics: 

-  DR  discovery  &  resolution 

-  Operating  time  (periodic  &  cumulative) 
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Summary 

DEVELOPMENTAL  TEST  &  EVALUATION 

•  Product  of  T&E  is  knowledge  for  decisions  across  the  life  cycle 

•  Value  of  T&E  -  informed  decisions  (acquisition  &  operational) 

•  No  single  set  of  metrics  applicable  to  all  decisions  or  phases 

•  Metrics  assess  how  well  T&E  is: 

-  Planning 

-  Executing 

-  Evaluating 

-  Reporting 


T&E  -  From  Concept  to  Combat 


15 


Next  Steps 


•  Engage  with  T&E  and  program  management  communities 

•  Continue  to  develop  &  evolve  metrics 

•  Request  your  inputs  to  make  the  metrics  meaningful  &  useful 
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Contact  Info 


Darlene  Mosser-Kerner 


Visit  our  website: 


Contact  us  to  provide  feedback  and  share  your  experience 
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Back-up 


DEVELOPMENTAL  TEST  &  EVALUATION 


Metric  Examples 


u 

N 

F 

A 

V 
O 
R 
A 
B 
L 

£. 

F 

A 

V 
O 
R 
A 
B 
L 
E 


x  Actual 

o  Projected  at  Completion 
DRU  ■  Design  Review  Update 


5000 


4500 


4000 


4498  .  TGT  (PROP) 
4295  GOAL  (DTW) 


— r 

JAN 


I 

JUL 


- 1 — 

JAN  93 
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“New  . . .  Improved" 

Test  &  Evaluation  Master  Plan 

Ms.  Darlene  Mosser-Kerner 

Developmental  Test  &  Evaluation 
OUSD(AT&L)/Systems  &  Software  Engineering 


New  TEMP  Content  &  Format 


•  Current  TEMPs  have  become  bloated  bureaucratic  packages 

-  Excessive  detail 

-  Late  to  need  -  frequently  completed  after  testing  has  started 

-  Limited  discussion  of  evaluation 

-  Allowed  “stovepiping”  within  T&E  community 

•  Need  to  improve  TEMP  relevance,  utility,  and  timeliness 

-  Focus  on  evaluations 

-  Facilitate  integrated  testing 

-  Show  support  for  Acq  Strategy  &  SE  linkage 

-  Elevate  discussion  level  to  T&E  strategy 

•  New  TEMP  Content  &  Format  in  DAG  update 
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Revised  TEMP  Concept 


DEVELOPMENTAL  TEST  &  EVALUATION 
Part  I  Part  II 

Introduction  Mgmt  &  Sched 


Brief  mission  description 
paragraph 

System  description 

Brief  Threat  Assessment 

Program  Background 

Key  Capabilities 


Describe  T&E 
management 

Common  Data 

Deficiency  Reporting 

TEMP  Updates 

Overarching  integrated 
schedule  that  includes 
sequencing 
of  T&E  activities 
(CT,  DT,  OT,  LFT,  M&S) 


Part  III 

T&E  Strategy 

The  philosophy  recognizes 
a  T&E  continuum  & 
emphasizes  evaluations 

Evaluation  Framework  ties 
T&E  knowledge  to 
decisions,  requirements,  etc 

Developmental 

Live  Fire 

IOT&E  Readiness  Cert 
Operational 


Part  IV 
Resources 

Include  in  para  form  or  table: 

•Test  articles  needed/event 
•Special  equip/  instr  costs 
•Target  /  expendable  costs 
•Threat  representation  costs 
•Manpower  needs 
•M&S  costs 


Certifications 


What 


Reliability  Growth 


Future  Testing 

Linkage  of  decisions  to  evaluations,  requirements,  test  phases,  and  resources 

Who,  When  Why,  How  Resources  required 

Include  Joint  requirements  throughout 
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Test  Planning  Hierarchy 


DEVELOPMENTAL  TEST  &  EVALUATION 

Scope 

System  Life  Cycle 


Acquisition  Phase 


Test  Type 


Test  Missions 


Individual  Test  Event 


Current  vs  New  Outline 


DEVELOPMENTAL  TEST  &  EVALUATION 

Current 

PART  I:  SYSTEM  INTRODUCTION 

>  Mission  Description 

>  System  Description 

>  System  Threat  Assessmen 

>  Measures  of  Effectiveness 
Suitability 

>  Critical  Technical  Parameters 


New 

PARTI:  INTRODUCTION 

1.1.  Purpose 

1.2.  Mission  Description 
System  Description 
•  Sys  Threat  Assessment 
1  Program  Background 
■  Key  Capabilities 


PART  II:  INTEGRATED  TEST 
PROGRAM  SUMMARY 

>  Integrated  Test  Program  Schedj 

>  Management 


PART  II:  T&E  PROGRAM 

MANAGEMENT  &  SCHEDULE 

2.1.  T&E  Management 

2.2.  Common  T&E  Data  Base  Requirements 

2.3.  Deficiency  Reporting 

2.4.  TEMP  Updates 

2.5.  Integrated  Test  Program  Schedule 
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Sample  Integrated  Schedule 


DEVELOPMENTAL  TEST  &  EVALUATION 


Fiscal  Year 

94 

95 

96 

97 

98 

99 

00 

01 

02 

03 

04 

05 

06 

07 

08 

09 

10 

11 

12 

Quarter 

ii  i  3  4 

ll  2  3;  4 

ll  2  3  4 

ll  2!  3;  4 

12  3  4 

1  21  3  4 

ll  2  3  4 

12  3  4 

12  3  4 

1  2  3I4 

12  3  4 

1 1  2  3  4 

12  3  4 

1  2  3:4 

1  2!  3!  4 

12  3  4 

ll  2  3  4 

1  2  3  4 

1  2  3:4 

"Reauirement 

£ 

cddD 

CPD 

U 

IOC 

★ 

t!Yf 

oc 

I 


E 


Technology  Development 

Acquisition  Milestones 


MS  B 


Engineering  and  Manufacturing 
Development  &  Demonstration 


o 

MPA  3RR 


MS  C 


Production  &  Deployment 

O'  FRP 


LRIP  /  IQTE 


FRP  DR 


Operations  &  Support 


Systems  Engi 


neering® 


SRR 


4 

SF 


V  ! 

R  PDR 


V 

CDR 


FRR 


0 

PRR 


p 


MSD/ 

|og 


oaistics  Events 


LA 


Vila 


ilaV 


ioc 


srV 


0  Core  Capability 


V 

Prototype 

Major  Contract  Evlents 


V 

EMDD 

01 

IBR 


r  LRIFt  Lot  1  / 

V  LRIP 


I0TE  support 


Lot  2 


rULeaiJ  Loti 


'f 


LRIP 


Lot  3 


L6 


»  LqI 

U\  FRP 

V 


Lot4_ 

V  FRP 


MYP 


Production 


EMDD 

Initial  T 

o  o  o 


LRIP 


UJL 


L/Leadl  Lot  2  O 


ead 1  Lot  3 


FRP  L/Lead  Lot  4 


ID 


L/Lead 


I  nt  5 


jl21 


L/Lead 


Lot  6 


L/Lead 


Lot  7 


L/Lead 


jl24! 


Lot 8 


x  24 1 


L/Lead  I  Lot  9  xlQO 


Training  Systems 


1 

TDFA  1 

TEk 

ng  (T&El  □ 


Flight  SimO  <>Maint.  T  rainers 
IpT  Training 


3TES 


TEMP 


D:  i 

□  OA 


II  Constructive  M&S 


Test  &  Evaluation 


yirt.uaJMSff  J 

I  SIL  I 


Constructive  M&S 


Virtual  M&S 


o  LF’  E 


SIL 


STEMP 
BLRIP 

Report 


E 


HITL 


.  ISIF 


I  LFTE 


[Components)  ll  LFTE  (Svstemsll 


IT1 


Report 


AC 

tR/iotf 

— 1 

IOT&E 

Mi: 

hMQ 

FOTE 


DIACAP 


Phas  i 


I  Definition  ,  .  .  .  .  . 

Phase  II  Verification  and  Certification  Testir 


IATT 


Phase  III  Validation  /  Cert  Tests 


Phase  IV  Post  Accreditation 


Funding 


RDTE 
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Current  vs  New  Outline 


DEVELOPMENTAL  TEST  &  EVALUATION 


Current 

PART  III:  DEVELOPMENT  TEST  AND 
EVALUATION  OUTLINE 

>  Development  Test  and  Evaluation 
Overview 

>  Future  Developmental  Test  and 
Evaluation  Limitations 

PART  IV  OPERATIONAL  TEST  AND 
EVALUATION  OUTLINE 

>  Operational  Test  and  Evaluation  Overview 

>  Critical  Operational  Issues 

>  Future  Operational  Test  and  Evaluation 
Limitations 

>  Live  Fire  Test  and  Evaluation 


New 

PART  III:  T&E  STRATEGY 

3.1  Introduction 

3.2  Evaluation  Framework 

•  Evaluation  Framework  Matrix  (Annex) 

3.3  Developmental  Evaluation  Approach 

•  Mission  Oriented  Context 

•  Test  Objectives 

•  M&S 

•  Test  Limitations 

3.4  Live  Fire  Evaluation  Approach 

•  Test  Objectives,  M&S,  Limitations 

3.5  Certification  for  IOT&E 

3.6  Operational  Evaluation  Approach 

•  Test  Objectives,  M&S,  Limitations 

3.7  Other  Certifications 

3.8  Reliability  Growth 

3.9  Future  Testing 
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Example  Evaluation  Framework 


DEVELOPMENTAL  TEST  &  EVALUATION 


Key  Requirements  and  T&E  Measures 

Test  Methodologies/Key  Resources 
(M&S,  SIL,  MF,  ISTF,  HITL,  OAR) 

Decisions 

Supported 

Key 

Reqs 

COIs 

Key  MOEs / 
MOSs 

CTPs  & 
Threshold 

Combat 
Radius 
KPP#1 : 

COI  #1.  Can  the 

UAV  locate 
and  engage 
the  XXX 
enemy  threat 
at  a  range  and 
time  that  will 

ensure 

survivability  of 

friendly 

troops? 

MOE  1.1. 

Range 

Fuel 

Consumption 

Aero  +  Propulsion  M&S 

Engine  stand 

Performance  profiles  -  OAR 

PDR 

CDR 

MS-C 

MOE  1.2. 

Speed 

Airspeed 

Wind  Tunnel 

Performance  M&S 

Performance  Fit  Test  -  OAR 

PDR 

CDR 

MS-C 

COI  #2.  Is  the 

XXX  suitable 
for... 

MOE  1.3. 

Post-CDR 

FRP 

KPP  #2 

MOS  2.4. 

Data  link 

MS-C 

SR 
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Current  vs  New  Outline 


DEVELOPMENTAL  TEST  &  EVALUATION 


Current  N6W 

PART  V  TEST  AND  EVALUATION  PART  IV:  RESOURCE  SUMMARY 

RESOURCE  SUMMARY  4.1  introduction 


> 

> 

> 

> 

> 

> 

> 

> 

> 

> 


Test  Articles  - 

Test  Sites  and  Instrumentation  — 
Test  Support  Equipment  — 

Threat  Representation  — 

Test  Targets  and  Expendables  — 
Operational  Force  Test  Support  — 
Simulations,  Models,  and  Test  Beds- 
Special  Requirements 


Test  and  Evaluation  Funding  Requirements 
Manpower/Personnel  Training  _ \  42 

4.3 

4.4 


•  Test  Articles 

•  Test  Sites  and  Instrumentation 

•  Test  Support  Equipment 

•  Threat  Representation 

•  Test  Targets  and  Expendables 

•  Operational  Force  Test  Support 

•  Models,  Simulations,  and  Test  Beds 

•  Joint  Operational  Test  Environment 

•  Special  Requirements 

Federal,  State,  Local  Requirements 
Manpower/Personnel  Training 
Test  Funding  Summary 


•  Resource  Summary  Matrix 
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Critical  Technical  Parameters 


DEVELOPMENTAL  TEST  &  EVALUATION 


•  CTPs  are  not  well  defined  or  productively  implemented 

•  A  short  review 

-  What  are  they? 

-  How  should  they  be  determined? 

-  How  should  they  be  used? 
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Critical  Technical  Parameters 


Definition 


DEVELOPMENTAL  TEST  &  EVALUATION 


•  Pick  the  CTPs  - 


^-Er 


adar  Target  Locations. 
{Tor  _ 

Interoperability 

MTBF 

Software  Functionality 


•  Support  Internet 
Protocol 


Range  Safety 
osition  Accurac^> 
Operational  Availability 
Critical  field  length 
Jammer  Duty  Cycle; 
Range 


•  Single  Mission  Sortie 

•  Open  Architecture 
Certification 

•  Interoperability 
Certification 

•  Handling  Qualities 


•  Definition:  A  CTP  is  a  measurable  critical  system  characteristic 
that,  if  not  achieved,  preclude  the  fulfillment  of  desired  operational 
performance  capabilities. 

•  CTPs  are  technical  measures  derived  from  desired  user  capabilities. 

•  CTPs  are  NOT  a  percentage  of  KPPs! 
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Critical  Technical  Parameters 

How  Derived? 

DEVELOPMENTAL  TEST  &  EVALUATION 

•  CTP  development  process  is  the  responsibility  of  the  program 
test  manager 

*  Lead  Systems  Engineer  plays  a  key  role  in  determining  CTPs 
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Critical  Technical  Parameters 


DEVELOPMENTAL  TEST  &  EVALUATION 


How  Used? 

■ 


*  While  not  user  requirements,  CTPs  are  technical  measures 
derived  from  desired  user  capabilities. 

*  Testers  use  CTPs  as  reliable  indicators  that  the  system  is  on 
(or  behind)  the  planned  development  schedule  or  will  likely  (or 
not  likely)  achieve  an  operational  capability. 

*  CTPs  should  be  significant  from  a  T&E  program  perspective  - 
should  drive  scope  /  magnitude  of  the  T&E  program. 
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New  Terminology 

DEVELOPMENTAL  TEST  &  EVALUATION 

•  Mission-oriented  context: 

>  Ability  to  relate  evaluation  results  to  an  impact  on  the  warfighters’ 
ability  to  execute  their  tasks 

>  More  robust  test  environment  allows  ID  of  design  issues  that  may 
not  be  discovered  in  a  pure  DT  environment 

>  Opportunity  to  influence  design,  increase  reliability,  performance 

•  Integrated  Testing: 

“Integrated  testing  is  the  collaborative  planning  and  collaborative 
execution  of  test  phases  and  events  to  provide  shared  data  in 
support  of  independent  analysis,  evaluation,  and  reporting  by  all 
stakeholders  particularly  the  developmental  (both  contractor  and 
government)  and  operational  test  and  evaluation  communities” 


- Oriented  Context 


Mission-oriented  DT&E  is  not  a  dress  rehearsal  that 
is  conducted  just  prior  to  IOT&E.  It  is  the  focus 
throughout  the  DT  program  to  ensure  the  design  of 
the  system  will  meet  the  user’s  needs. 


•  Part  of  policy  to  emphasize  robust  DT&E 

>  Discover  operational  failure  modes  in  time  to  fix  them 

•  Mission-oriented  DT  and  Integrated  Testing  will  increase 
efficiencies  and  reduce  risk 


Bonus  -  New  TES  Sneak  Peak 

DEVELOPMENTAL  TEST  &  EVALUATION 

•  T&E  Strategy  required  at  Milestone  A 

•  TEMP  format  -  4  parts 

•  Less  detail  -  similar  to  “draft”  TEMP 

•  Includes  T&E  life  cycle  concept 

•  Includes  TDS  test  plan 
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•  New  TEMP  Content 

-  Brings  evaluation  focus  into  TEMP 

-  Assumes  a  continuum  of  T&E 

-  Life  cycle  view  versus  scoping  to  next  milestone 

-  Facilitates  Integrated  Testing  &  Mission-oriented  context 

-  Additional  test  plan  details  shifted  to  System  Test  Plan 

•  In  next  revision  to  DAG  -  Chapter  9 

-  Applies  to  new  programs,  restructured  programs,  &  others 
if  desired 
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Contact  Info 


Darlene  Mosser-Kerner 


Visit  our  website: 


Contact  us  to  provide  feedback  and  share  your  experience 


Carnegie  Mellon 

Software  Engineering  Institute 


NDIA  2008  Systems  Engineering  Conference 


Building  net-ready  information  interoperability 
performance  indicator  widgets  for  DoDAF  2. 0 

dashboards 


Jayson  Durham 

NAERG/ELS  Project  Lead  (ASN  RDACHSENG) 
SPAWAR  Systems  Center  Pacific,  Code  56150 
iavson.durham@navv.mil 

Bill  Anderson  &  David  Zubrow 
Carnegie  Mellon,  Software  Engineering  Institute 
wba@sei.cmu.edu.  dz@sei.cmu.edu 


Carnegie  Mellon 

__  Software  Engineering  Institute 

Agenda 

Motivation 
Goal  Driven  Measurement  -  GQIM 
Workshop  Outcomes 
Case  Example:  Mission-Architecture  IPT 
Next  Steps 


2 


HSDII  Committee  Objective 


r^J^^Group 


Benefit  ITAA/GEIA  members ,  government 
sponsors,  builders,  developers,  and 

users  of ... 

Products,  Processes  and  Tools  related  to  . 

Information  Interoperability  by ... 

Filling  critical  gaps, 

Improving  performance,  and 
Reducing  costs. 


^I^Group 


But  How  Do  We  Judge? 


VrfgQra.1 

•  V 

■  T: 

ftaA 


Carnegie  Mellon 

__  Software  Engineering  Institute 

Agenda 

Motivation 
Goal  Driven  Measurement  -  GQIM 
Workshop  Outcomes 
Case  Example:  DODAF  2.0 
Next  Steps 
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Carnegie  Mellon 

Software  Engineering  Institute 


Goal-Driven  Measurement 

When  using  goal-driven  measurement, 
the  primary  question  is  not: 

“What  metrics  should  I  use?" 

rather,  it  is: 

“What  do  I  want  to  know  or  learn?” 
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Measuring  Goal  Achievement 


Success 
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Analysis 
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Strategy  to 
accomplish 
goal 
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Tasks  to 
accomplish  goal 
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Indicator  Template 


Workshop  Steps 


Indicator  Template 

Goal  ID:  - 

Objective  _ 

Question  _ 


Dpi 


Inputs 

Algorithm 

Assumptions 


Step®:  Goals 


Components  of 
good  goal  statements 

I 


Step(2): 

Clarify  Questions 
to  refine  the  goal 


Step® : 

Decomposing 

Goals 


Subgoals  by 
perspective 


Step  (5) : 
Operationalize  Goals 

Operationalize 
goal  statement 


Step  (?) : 

Strategies  & 
Activities 

Step  (6) :  Success  Indicators 

Postulate  Success  Indicators 

Step®:  Success  Criteria 


Clear  articulation  of 
the  criteria  you  will 
use  to  decide  if  the 
goal  has  been  met. 


Step  (8) : 

Identify  the  data  elements 
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needed  to  implement 
your  measures 
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Step  ^0):  Prepare  a  plan 


Verification  and 
action  plans 
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Workshop  Outcomes:  Top  Three  Goals 


Enable  precision  information  sharing  among  stakeholders 

-  Minimal  ambiguity 

Measure  the  “goodness”  for  information  interoperability 
standards 

-  Then  standards  in  general 

-  “Goodness”  for  information  interoperability 

-  How  effectively  are  users  getting  &  using  information 
exchanges 

Systems  and  enterprise'^  achieve  more  effective  collaboration 
and/or  achieve  greater  success  by  enabling  inter-enterprise 
collaboration 


Enable  information  sharing  among 
stakeholders  with  minimal  ambiguity. 


^I^Group 


f-W-in 

•  V 

■  T: 

fTAA 


Type  of 
Indicator 
Theme  {Success, 

{Stakeholder  Progress,  Atomic1  Roll-up  DoDAF  Product  &  Other 

ID  ,  Info,  Std)  Analysis)  Questions  Indicator  Indicator  Sources  CADM  Data  Elements 


Stakeholder 

Who  are  the  stakeholders? 

OV-2,  OV-3 

OperationalNode 

Stakeholder 

What  information  do  the  stakeholders 

need? 

Completeness 

Is  there  an  information  architecture  that 

defines  the  stakeholders  and  context  for 
the  information  sharing? 

Completeness 

What  percentage  of  information 
exchanges  (lERs)  are  defined  in  the 
information  architecture? 

OV-2,  OV-3 

Information  Exchange  Reg  uiremen 
t / 

InformationExchange/ 

Needline 

11 

Information 

Success 

Was  the  correct  information  provided  where 

and  whan  needed? 

Visibility 

Did  the  Info  Provider  publish  availability 
of  the  info? 

Can  the  Info  Consumer  discover  needed 

info? 

Visibility 

What  percentage  of  lERs  are 
published? 

SV4a/b.  Enterprise  Catalog 
Service  (ECS)  &  Service 
Registry,  Content  Discovery 
and  Delivery  (CD&D) 

1  nfo  rm  at  i  o  n  Exc  h  a  ng  e/ 

Information  Element 

12 

Information 

Success 

Does  the  info  exchange  enable  traceability 

back  to  the  original  context 

Understandable 

Is  metadata  published  with  the 
information  that  defines  its 
s  o  u  rce/c  o  nt  ext  /  p  ed  iq  re  e  ? 

Understandable 

What  percentage  Info  Providers 
create  and  publish  metadata  for 
lERs/Services? 

AV-2,  SV-1 1 ,  DDMS, 
Metadata  Registry  [MDR) 

InformationExchange/ 

Information  Element 

13 

Information 

Success 

Can  we  verify  information  integritii? 

Unambiguous 

Does  the  Info  Provider  claim/advertize 

that  the  information  conforms  to  a 

'verifiable'  standard? 

Unambiguous 

What  percentage  of  lERs  conformed 
to  adopted  Standard(s)? 

TV-1,  MDR 

Information  Element,  Operational 
Nodes,  Technology  Areas, 
Technical  Standards. 

Performance  Parameters 
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Information 

Progress 

Is  the  information  standardised? 

Standard 

Analysis 

Are  technical  requirements  in  the  standard 

stated  clearli)  and  understandable? 

Extensible 

Does  the  standard  contain  normative 

statements  that  define  conformance  to 
compliance  points? 

DISR,  TV-1 

Technology  Areas,  Technical 
Standards,  Performance 
Parameters 

Standard 

Analysis 

Are  the  requirements  in  the  standardfs) 

implementable? 

Implementable 

Do  normative  statements  in  the 

standards  conflict? 

EMSR,  TV-1 

Technology  Areas,  Technical 
Standards,  Performance 
Parameters 

Standard 

Analysis 

Does  the  standard  define  aerification  of 

conformance? 

Testable 

Are  the  normative  statement  verifiable? 

DISR,  TV-1 

Technology  Areas,  Technical 
Standards,  Performance 
Parameters 

Standard 

Will  the  standard  support  sharing  with 

unanticipated  stakeholders? 
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Goodness 


Quality  Evaluation 


^|^Group 


•V 

ITAA 


Standard  Quality  Evaluation 


Categories  I - 

Notes: 

I  Each  category  is  graded  on  a  scale  of  1  -5  and  weighted  to  a  total  of  1 00%. 
Data  is  based  on  a  survey  of  stakeholders. 

I  Users  of  the  indicator  include: 

I  •  standard  developers  and  associated  marketing 
I  •  potential  adopters 
I  •  actual  users 

Scalability  means  across  multiple  domains 
I  Usability  means  by  multi-functions  (non-IT  experts) 
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Dollars 


^^LcBroup 


Cost  of  Interface  Maintenance 
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Latency 


Session  time 


Session  time 


Group 


DoDAF  2.0  “Dashboards” 


Vr«g»ra.i 

-•  V 

■  T 

ITAA 


Presentation  Techniques 


Architecture  Information 


Process  NVodeNs 

OaXa  VAodeVs 


VwrvcVvorv 


Svsxerrv 


Architecture  Methods 


Defining  indicator  widgets  for  dashboards 
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Workshop  Outcomes:  Conclusions 


The  SEI  GQ(I)M  provides  a  viable  methodology  to 
develop  information  interoperability  indicators 

We  identified  a  preliminary  set  of  indicators  for 
measuring  the  “goodness”  of  information  exchange 
standards  relative  to  business  goals 

We  concluded 


•  Enterprise  architecture  frameworks  with  an 
explicit  focus  on  services  (transactions)  provide  a 
means  of  implementing  and  improving  Information 
Interoperability 

•  Indicators  provide  a  means  for  establishing  a 
standardized  set  of  reusable  dashboard  elements 
(..indicator  widgets")  in  these  frameworks 


!AGr°uP  \S*-, 

Workshop  Outcomes:  Observations 

The  DoDAF  2.0  presentation  technology  working 
group  has  set  forth  dashboards  as  a  category  of 
presentation  views 

Baseline  indicators  for  information  interoperability 
need  to  be  developed  (similar  to  baseline  KPI"s  for 
enterprise  architecture  frameworks) 

Existing  work  from  assessment,  performance,  and 
other  model  based  efforts  provide  valuable 
resources  for  developing  information 
interoperability  (as  well  as  other)  indicator  widgets 
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Architecture  Models:  Background 
Tenets  of  Network  Centric  Warfare  (NCW) 


Eda 

Chief 

Bystems 

Engineer 


(Networked  Force,  Quality  Information,  Information  Sharing,  etc.) 


^  Precise''' 
Application 
.  Of  Force  > 


individual 

Situational 

Awareness 


Precision 

Effects 


Mission 

Effectiveness 


Common 

Tactical 

Picture 


^Shared 

Situational 

Awareness 


Robustly 
Networked 
t  Force  > 


Seif  > 
Synchronization 


Information 

Sharing 


Collaboration 


Tempo 


Speed  of 
Maneuver 


New  Concepts 
.  &  TTP  ^ 


Quality  of 
Organic 
Information 


Quality  of  Organic  and  Shared 
Information  is  The  Primary 
Focus  of  Terminology  and 
Lexicon  Services 


Quality  ol 
Shared 
Informatio 


Lexicon  Services 

Information 
Domain 


Cognitive 

Domain 


Physical 

Domain 


Source:  “Fighting  The  Networked  Force,”  Network  Centric  Warfare  2005,  John  J.  Garstka,  27  January  2005 
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Terminology  Services:  Challenge 
Semantic  Interoperability  Scoping 


Eda 
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Bystems 

Engineer 


Strong 

Semantics 
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Some  Intelligence,  Defense, 
Security,  Health,  Science  & 
Business  applications  shore 
information  at  these  levels 


DRM  1.5  sets 
the  bar  here 


Logical  Theory 


Axiology 
Modal  Logic 
First  Order  Logic 


Conceptual  Model  UML 

■  y 

^  /  RDF/S 

Topic  Map 


Description  Logic 
OWL 

Semantic  Interoperability 


Taxonomy 


Structural  Interoperabifity 


ER  Model 

DB  Schema,  XML  Schema 


Thesaurus 


Syntactic  Interoperability 


Relational  Model,  XML 

Many  Federal  applications 

Glossary 

Controlled  Vocabu  lary 

do  not  enable  data  sharing 

femonfics 

Terminology  Services  Discovery  Intelligence  Question  Answering  Reasoning 

Primary  Focus  Increasing  Search  Capability 
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Terminology  Services:  Challenge 
Information  Interoperability 


Eda 

Chief 

Bystems 

Engineer 


Common  Lexicon  vs  Ad-Hoc  Reverse  Engineering 


Reverse  Engineering 
Ad-Hoc  Terminology 


Reverse  Engineering 
Ad-Hoc  Terminology 


Reverse  Engineering  is  Expensive,  Difficult,  and  Often  Not  Feasible 


22 


Terminology  Services:  Related  Work  Chief 

Capability-Based Systems-of-Systems Engineering (SOSE)  £™ 

NCEE  NAERG 


Naval  Collaborative  Engineering  Environment  (NCEE) 
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Architecture  Frameworks:  Current  Work 

Mission  Architecture  Dashboard  systems 

Engineer 
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Validation,  and  Gap  Analysis 


Enterprise 
Framework  - 
Policies,  Rules,  Metrics 
Processes,  Architectures, ... 

M«dfcral  ikulUal  A»arrMM  to  ikv  TWaler 

i9i« i  . 


f  Inter-Agency 
Cross-Domain 
Information-Sharing 
“■  Supply-( 


^Mission  Architecture^ 
Dashboard 


Operational 

Mission-Threads 

r 

♦  • 

Operational  Plans  (OPIans) 
Mission  Operations 


Systems 

Devices 


Architecture  Models:  Emerging  Standards 
DoD  Architecture  Framework  2.0  Example  Systems 

Engineer 


Standardized  Dashboard  Views,  Data  Elements,  Business  Rules,  etc. 
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Architecture  Frameworks:  Emerging  Focus 

-  -  *  •  Qhief 

Bystems 


Enterprise  Dashboards  and  Widgets 


Engineer 


Roles  & 
Responsibilities 


KPI  -  Key  Performance  Indicator 
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Dashboards 


Multiple 
Dashboards  & 
Decision  Makers 


<£>  Presentation  Layer 
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Data  Resources 
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Group 


Way  Ahead 


Further  explore  leveraging  DoDAF  2.0  to 
help  define  an  open  standard  for  indicator 
widgets 

Produce  an  exemplar  reference 
implementation  for  a  set  of  indicator  widgets 

Produce  guidance  on  how  to  go  from 
existing  standards  to  the  indicator  widget 
paradigm 


Carnegie  Mellon 

Software  Engineering  Institute 


Questions? 
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IJ^Group 


Resources  Related  to  Information  Interoperability 
Indicator  and  Assessment 


•  V 

■  T: 

IT  A  A 


•  DOD  Net-Centric  Checklist,  Version  2.1.4,  July  30,  2004 

-  Assists  program  managers  in  understanding  the  net-centric  attributes  that  their  programs  need  to 
implement  to  move  into  the  net-centric  environment  as  part  of  a  service-oriented  architecture  in  the 
Global  Information  Grid. 

•  NCIOC  Network  Centric  Analysis  Tool  (NCAT)  &  SCOPE  model 

-  NCAT  is  a  metric  measurement  tool  developed  by  the  NCOIC  for  use  in  evaluating  the  ability  of  a 
system/subsystem/component  to  operate  in  a  network  centric  environment.  Designed  to  leverage 
complementary  tools  developed  by  DISA  and  others,  the  NCAT  is  highly  flexible,  easily  adaptable,  and 
can  be  tailored  for  specific  requirements. 

•  DOD's  Modular  Open  Systems  Approach  (MOSA)  Program  Assessment  and  Rating  Tool  (PART) 

-  An  analytical  tool  to  aid  DoD  Program  Managers  assess  their  approach  to  open  systems  throughout  the 
acquisition  life  cycle. 

•  Navy  Open  Architecture  Assessment  Tool  (OAAT) 

-  A  Navy  tool  to  assess  the  openness  of  a  systems  or  program. 

•  DOD's  Data  and  Service  Exposure  Verification  Tracking  Sheets 

-  Used  to  measure  net-centricity  in  support  of  the  DOD's  Net-Centric  Data  Strategy. 


A  catalog  of  reusable  indicators  can  be  readily  derived 
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•  Purpose:  to  introduce  a  simulation  based  methodology  to 
facilitate  development  of  a  software  product  line  architecture 
concept  for  the  Navy ’s  C5ISR  systems. 

•  Two  key  advantages  to  the  proposed  methodology: 

1 .  it  provides  a  formal  systems  approach  to  the  verification  of  the  product 
line  architecture  requirements  consistent  with  the  Department  of 
Defense  Architecture  Framework. 

2.  it  provides  a  medium  for  the  iterative  development  of  architectures 
that  blend  the  operational  concepts  of  FORCEnet  with  the  system  and 
technical  imperatives  of  Open  Architecture  and  Services-Oriented 
Architecture  (SO A). 


Sensor  Grid 
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What  I’m  Going  to  Tell  You 


Background 

Technical  Approach 

-  Key  Concepts 

-  Open  Architecture 

-  Domain  Modeling 

-  Formal  Methods 

-  H-P  Method 

-  Details  of  the  T echnical  Approach 

Conclusion 
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Background 


•  The  last  1 5  years  (or  thereabouts)  has  seen  a  number  of 
interesting  developments  in  the  technologies  that  support 
C4ISR  system  development. 

-  For  example,  the  advent  of  CEC  and  GPS  provided  the  impetus  for  the 
conceptual  development  of  Network-Centric  Warfare  (NCW), 
Network-Centric  Operations  (NCO)  and  FORCEnet  [Alberts,  Garstka, 
and  Stein  2000]. 

-  Yet,  despite  all  that  has  been  written  about  the  concepts  of  FORCEnet 
and  Open  Architecture  (OA),  there  has  been  little  written  on  how  these 
two  concepts  will  come  together  in  the  naval  C4ISR  systems  of  the 
future. 

•  The  main  emphasis  has  been  on  technologies  such  as  Internet 
Protocol  version  6  (IPv6),  not  the  architecture. 

•  As  a  result,  there  is  no  commonly  shared  or  understood 
model  of  what  this  end  state  may  look  like. 
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There  is  a  tendency  to  view  the 
system  architecture  using  existing 
paradigms  that  were  used  to 
develop  the  “stove-piped” 
systems  that  are  now  proving  to 
be  limited  in  their  capability. 

This  is  a  “paving  the  cow  paths” 
approach  and  has  made 
developing  FORCEnet  capable 
systems  difficult. 


ifrV. 


European  firms  such  as  Thales, 
Saabtech  and  Terma  have  already 
validated  the  concepts  of  open 
architecture,  software  product 
lines,  and  software  reuse  as 
applied  to  combat  systems 


More  Background 
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Key  Concepts 


•  In  addition  to  lessons  learned  from  European  firms, 
the  proposed  Technical  approach  is  built  upon  lessons 
learned  from  Lockheed  Martin’s  Norwegian  Frigate 
Project  and  a  predecessor  program,  Taiwan’s  PFG-2 
Class  Frigate  project 

•  Valuable  lessons  were  also  learned  from  the 
predecessor  program  to  OA,  the  Common  Command 
and  Decision  (Common  C&D)  project. 

•  Common  C&D  resulted  in  the  development  of  several 
FORCEnet  related  concepts  that  were  briefed  to  the 
Assistant  Secretary  of  the  Navy  for  Research  and 
Development. 
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OA  Principles 


•  The  key  Open  Architecture  principles  espoused  by  the  Navy 
are  [Naval  OA  Strategy]: 

-  Modular  design  and  design  disclosure 

-  Reusable  application  software 

-  Interoperable  joint  warfighting  applications  and  secure  information 
exchange 

-  Life-cycle  affordability 

-  Encouraging  competition  and  collaboration  through  development  of 
alternative  solutions  and  sources 

•  The  first  two  principles  are  especially  relevant  to  this  paper.  It 
is  the  authors’  belief  that  proper  attention  to  these  principles 
will  result  in  software  product  lines  that  provide  domain 
specific  solutions. 
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•  The  ability  to  make  good  design  decisions 
early  in  the  process  is  a  significant  driver  in 
effectively  lowering  life-cycle  cost  and  system 
development  time. 

•  There  are  two  key  issues  to  be  addressed  with 
the  use  of  the  Open  Architecture  concept: 

-  What  is  the  structure  of  the  various  product  lines 
required  to  support  the  various  warfare  domains, 
and 

-  What  is  the  technical  approach? 
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Domain  Modeling 
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Formal  Methods 


•  Formal  methods  are  mathematically-based  techniques  for  the 
specification,  development  and  verification  of  software  and 
hardware  systems. 

•  Natural  language  specifications  tend  to  get  out  of  hand  as  the 
document  grows  and  with  growth  comes  ambiguity. 

•  The  use  of  formal  methods  for  software  and  hardware  design 
is  motivated  by  the  expectation  that,  as  in  other  engineering 
disciplines,  performing  appropriate  mathematical  analyses  can 
contribute  to  the  reliability  and  robustness  of  a  design. 

•  Formal  methods  are  appropriate  for  the  design  of  discrete- 
event  real-time  systems  because  they  can  be  used  to  specify 
system  behavior  without  ambiguity. 
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The  Approach 
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A  Petri  net  consists  of  places,  transitions,  and  directed  arcs 


Inputs 


The  following  approach  uses  two  formal 
methods  as  a  foundation: 

-  Finite  State  Machines  (FSM) 

-  Petri  Nets 
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The  Methodology 


•  Centered  around  the  Hatley-Pirbhai  “ Process  for 
Systems  Architecture  and  Requirements  Engineering" 
(PSARE) 

-  Model-based  process  that  uses  FSM  &  Petri  Nets 

-  Accommodates  HW,  SW  &  PW 

-  Can  be  described  using  SYSML/UML  or  EFFBD’s  (to 
name  two)  (not  tool  dependent) 

-  Results  in  both  a  functional  and  architectural  specification 
model 

-  Can  be  captured  with  Clymer’s  OpEMCSS  modeling 
approach  which  represents  both  FSM  and  Petri  Nets 

•  Core  elements  are  the  process/control  model  and  the 
architecture  template 

Operational  Evaluation  Modeling  for  Context  Sensitive  Systems 
http://www.ecs.fullerton.edu/~jclymer/ 
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H-P  Overview 


System  Architecture 


ACD 
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The  elements 


H-P  originally  used 
Yourdon-DeMarco 
notation 

The  steps 

Figures  used  with  permission 
from  H&A  Systems  Engineering 
http://www.hasys.com/ 


Figure  1  Model  development  process.  Enhancing  requirements  models  for  each  architecture  module 
before  drawing  flows  on  the  AFD  provides  consistent  allocation  of  architecture  flows  to  interconnects. 
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Allocating  to  HW,  SW  &  PW 


Figure  2  Hardware^software  interface  modeling  using  stores.  Hardware  processes  produce  flows 
into  stores  which  are  accessed  by  software  processes.  Software  processes  produce  flows  into  stores  which  are 
accessed  by  hardware  processes  and  transformed  for  intermodule  communication. 


Figure  used  with  permission 
from  H&A  Systems  Engineering 
http://www.hasys.com/ 
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Clymer’s  OpEMCSS  Approach 
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Figure  1 1 :  OpEMCSS  Directed  Graph  Model  of  a  Part  Production  System 


OPTIMIZING  PRODUCTION  WORK  FLOW  USING  OPEMCSS 
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H-P  Advantages 


Table  1  HPM  Features  and  Benefits.  The  rigor  and 
hierarchical  nature  of  HPM  provide  specific  benefits. 


Figure  used  with 
permission  from 
H&A  Systems 
Engineering 
http://www.hasys 
com/ 
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Features 

Benefits 

Hierarchical  model 

*  Specifies  requirements  at 
appropriate  level 

*  Depicts  manageable  amount 
of  information  at  one  time 

Graphical  and  text 
representation  of 
functionality 

*  Clearly  shows  interfaces 
(functional  and  physical) 

*  Graphics  depict  abstract 
aspects  of  system 

*  Text  defines  details 

Allocation  of  functions 
to  physical  entities 

*  Greatly  improved  interface 
consistency 

Rigorous  method 

*  Promotes  thorough  design 

*  Identifies  gaps  early 

WWW.NPS.EDU 
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•  Another  advantage  of  a 
simulation-based  approach  using 
H-P  can  be  seen  by  reference  to 
the  figure. 

•  As  system  development  proceeds 
down  the  left  side  of  the  “Vee” 
the  models  developed  provide  the 
foundation  and  guidance  for  the 
steps  as  integration  proceeds  up 
the  right  side  of  the  “Vee”. 

•  It  should  noted  that  the  “Vee” 
model  has  been  demonstrated  to 
be  consistent  with  spiral 
development 
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design 
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Software 

integration 

testing 
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software 
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Conclusion 


The  presented  work  gives  emphasis  to  the  value  of  a  formal  process  in 
architecture  development. 

In  this  case  formal  will  mean  that  the  architecture  requirements  will  be 
validated  through  the  use  of  simulation  as  part  of  a  defined  methodology  as 
described. 

Specifically,  the  model  driven  architecture  approach  has  the  following 
advantages: 

-  It  is  a  formal  method  for  tying  the  architecture  requirements  process  to  the 
architecture  verification  process. 

-  It  is  consistent  with  acquisition  policy 

-  It  provides  a  methodology  to  test  Network  Centric  Operations  concepts  such  as 
MDA,  CMD,  and  TCT. 

The  use  of  a  simulation-based  methodology  will  result  in  the  requisite 
DODAF  artifacts  required  for  both  requirements  capture  and  the 
description  of  the  system  functional  behavior. 

In  addition,  it  supports  the  development  of  architectures  that  incorporate 
modular  design  and  the  identification  of  reusable  and  interoperable 
modules/applications. 

This  approach  is  consistent  with  the  development  of  a  capability/systems- 
based  architecture  using  a  spiral  or  “Vee”  approach. 
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Future  Work 


Incorporation  of  the  use  case  paradigm 

Mapping  to  DoDAF 

Incorporation  of  Clymer’s  work 

Merging  notations/languages  into  a  universal 
architecture  descriptive  framework 
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Overview 


□  Introduction 

□  The  Process  Problem 

□  Tools  are  a  solution 

□  Examples 

□  Substantiating  Data 

□  Conclusion 


Introduction 


The  US  Army  Armament  Research 
Development  and  Engineering  Center 
(ARDEC)  Systems  Engineering  Directorate 
(SED)  Systems  Engineering  Infrastructure 
Division  (SEID)  has  a  completely  documented 
Systems  Engineering  Process. 


Products 


The  Process  Problem 


All  the  “best”  processes  in  the  world  are 
useless  if  they  are  not  accepted,  understood 
and  implemented  by  the  workforce 


Difficulties  with  Process  Acceptance 


□  Hard  to  understand/implement  the  process 

□  Don’t  know  what’s  available  to  help 
process  implementation 

□  No  common  method  of  implementation 

□  Uncertainty  on  the  part  of  the  user  and  the 
advocate  on  whether  implementation  is 
being  done  correctly. 


Tools  are  a  solution 


□  The  US  Army  ARDEC  Systems  Engineering  Directorate 

(SED)  has  been  investing  in  its  infrastructure  via  tools 

that  facilitate  proper  use  of  its  processes 

■  Many  are  simple  Excel/Access  tools  that  were  developed  in 
<100  man  hours 

□  Tools  that: 

■  Guide  the  user  through  the  process  and  document  the  results  of 
each  step  (DAR,  Peer  Review,  Roadmap) 

■  Evaluate  a  project’s  compliance  to  process(es)  (PP  Eval) 

■  Guide  the  user  towards  additional  resources  to  assist  them  (PP 
Eval,  IPPD,  PAL) 

■  Get  the  user  started  with  some  instruction  (Requirements 
Management  Plan  Template,  System  Spec  Template) 

■  Provide  the  user  with  examples  to  choose  from  (Technical 
Engineering  Database  (TED),  Example  Project  Plans) 

□  Feedback  has  shown  that  they  improve  process 

utilization 


ARDEC  SE  Roadmap 


The  SE  Roadmap  Tool  encompasses  17  ARDEC  SE 
process  areas  that  describe  key  aspects  of  SE  tasks 
covered  by  projects  during  the  complete  product  lifecycle 
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SE  Roadmap  Implementation 


Roadmap  provides  basis  for  technical  planning, 
feeds  the  Project  Plan/Schedule,  allows 
technical  assessment  versus  planned  activities 
and  supports  multiple  reporting  needs 
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Project  Schedule  reflects  SE 
activities  and  events  needed  to 
develop  technology/product  and 
satisfy  roadmap  transition  criteria 
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Roadmap  Implementation  Guide: 
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required  SE  End  State  (use  SE  procedures  to  assist  with  tailoring 
as  appropriate) 

Work  with  APO/IPT  to  determine  what  SE  Tasks  will  satisfy 
transition  criteria  to  achieve  next  SE  level  &  complete 
Roadmap  accordingly 

Develop  or  update  Project  Plan/Schedule  using  Roadmap 
input  to  complete  SE  sections  of  plan 

Verify  that  I  MS/Project  Schedule  reflects  accomplishments, 
schedule  and  products  contained  in  Roadmap 
Assess  Project’s  performance  against  Roadmap  details  and 
provide  status  in  the  Roadmap  Column  title  “status”.  Describe 
any  corrective  actions  as  required. 

Use  tech  assessment  from  Roadmap  to  address  reporting 
requirements  (MAPR,  Level  1,  Level  2,  Level  3  Briefings,  etc.) 


J>| 


Reporting  Processes 


¥1  PP.  n  fq  H  n*ii l  e 


MAPR  format  includes  SE  Status 
(tech  assessment),  project  rating, 
and  corrective  action  plan  (if 
needed) 

Level  1  Briefing  summarizes  SE  ^ 
status  of  project 


<Project  Name> 
Level  1  Review 

<date> 

PRESENTED  BY 


(Project  Name)  SE  Status 

SEL:  (Name) 


JSE  Planned  Time-  Associated  | Status 

[Activities_ frame  Product(s) 


SE  Roadmap  is  the  Linchpin  that  Ensures  Effective  Technical  Planning  and  Technical  Assessment  Activities  on  Projects 


Project  Plan  Evaluation  Tool 


□  The  Project  Plan  (PP)  is  a  key  piece  of  the  ARDEC 
Project  Management  process 

■  Originally  developed  for  Project  Plan  (PP)  evaluators  to 
perform  an  assessment  of  a  PP  which  lead  to  PP  approval 
and  project  funding 

■  Quickly  became  a  key  instructional  document  provided  to 
projects  who  were  writing/updating  their  PP 

■  Also  used  to  capture  the  Ownership  Matrix  for  every  PP 
section  (who  the  SME  is  for  each  PP  section).  This  provides 
key  contact  for  further  assistance 

■  Process  Flow 

■  Automatically  tailors  the  Evaluation  Criteria  based  on  project 
details  that  are  used  to  “seed”  the  tool,  (project  scenario, 
phase  etc.) 


Project  Plan  Evaluation  Tool 


□ate  Received: 

<Date  Received;- 

PI  Name: 

<Name  of  Pb 

PIO  Evaluator  Name: 

<  Enter  PI  Evaluator  Name  Here> 

SEAL  Name: 

<  Enter  SEAL  Name  Here;- 

Process  Advocate  Name: 

<  Enter  Name  Here;- 

Project  Name: 

EXAMPLE  Project 

PP  Version  #: 

<PP  Version  Number;- 

Project  Scenario: 

<PP  Scenario;- 

Project  Phase: 

Technology  Development 

APO  Name: 

<APO  Name;- 

The  Questions  are  tailored  based  on  the  Project  Details  above 


15  Sjslgm  Engineering  Process 

1.  Does  the  PP  describe  the  overall  SE  process  to  be  used  on  the  project  and  the  basis  for  selection 
(e  g  ,•  commercial  standaid,  organisational  process,  etc.),  including  the  purpose  and  objectives  of  the 
process? 

2.  Does  the  PP  describe  the  technical  authority  responsible  for  implementation  of  the  SE  process? 


T5.1  Requhoment  Development 

1.  Does  the  PP  specify  the  approach  and  methods  used  to  define  the  performance  and  functional 
requirements  (including  all  product  and  component  functional  requirements,  performance 
requirements,  interface  requirements,  and  other  detailed  requirements]  P 

Each  Project  Plan  section  is  evaluated 


Ownership  Matrix  details  SMEs  for  each  section 


_ Feedback _ 

We  welcome  feedback  on  the  Evaluation  Criteria  and  this  Evaluation  Tool  itself.  Please  use  the  link  below  to  send  feedback  to  Frank  J.  Salvatore 

(SED),  Humisar  Sianipar  (PIO)  and  Dan  Crowley  (PIMG). 


PP  Sect.  # 

Project  Plan  Section 

PIMG 

SED 

PIO 

19 

Interface  Management 

Rob  Bernard 

20 

Process  Assurance  Plan 

Process  Advocate 

21 

Configuration  Management 

Paula  Baselinesa-Versi 

22 

Data  Management 

Petro  Librariano 

23 

Project  Deliverables/Work  Products 

Project  Integrator 

24 

Simulation  Support  Planning 

Modello  Simulate  Jr. 

25 

Risk  Management  Plan 

Project  Integrator 

Process  Flow 


Click  to  Send  Feedback 


Decision  Analysis  and  Resolution 


□  Process  is  nested  within  the  tool 

■  Each  Process  Step  has  a  corresponding 
section  of  the  tool. 

□  Use  of  the  tool  provides  a  project  with  “self 
documenting”  input  data  and  results 

□  Provides  the  user  with  some  standard 
graphical  forms  of  output  that  assist  with  both 
making  the  decision  and  capturing  its  rationale 

□  Use  of  the  tool  follows  the  DAR  Procedure 


Decision  Analysis  and  Resolution 


Follow  the  process  steps  in  the  tool 


Idaho  National  Laboratory 


Criteria 

Weight 

Alt  1 

Alt  2 

Alt  3 

Alt  4 

Cost 

0.563 

30000 

50000 

25000 

80000 

Schedule 

0.100 

5 

3 

1 

7 

Safety 

0.150 

Good 

Fair 

Bad 

Great 

Startup 

Risk 

0.188 

4 

3 

9 

1 

Raw  Input  for  each  Alternative 


1 

Goal 

Number 

Goal  Name 

- * 

Goal  Weight 
Factor 

- 

Normalized 
Weight  Factor 

- ^ 

Criteria  Name 

Criteria  Weight 1 
(within  each 
goal) 

1 

Reduce  Cost 

0.563 

Cost 

10 

2 

Meet  Schedule 

4 

w 

0.250 

w 

Schedule 

2  ! 

r 

r 

Safety 

3  1 

3 

Reduce  Risk 

3 

r 

0.188 

r 

startup  risk 

s  ; 

Identify  and  Weight  Goals  +  Criteria 
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Utility  applies  to  the  Raw  Input  to  Score  the  Alternatives  against  the  Criteria 


Integrated  Process  &  Product 

Development  Tool 


□ 

database  of  available  resources  (procedures 
tools,  templates  etc.) 

□ 

earch  based  on  different  “languages”  (DOD 
lifecycle,  Six  Sigma,  SED  SE  Process...),  and 
the  sub-steps  within  that  language 


Integrated  Processes  and  Tools 

Help  You  Find  the  Best  Processes  and  Tools  to  Support  IPPD 


Use  Drop  Down  Lists  to  Generate  Report 


Process/Method  T 

Procedure/Step  y 

Systems  Engineering  Process 

Technical  Planning 

Resource 

Type 

User 

Purpose 

Reference 

Reference  Location 

POC 

Applies  to 

Work  Breakdown 
Structure  [WBS} 

Tool 

APO 

To  better  define  and  organize  the 
total  scope  of  a  project  using  a 
hierarchical  tree  structure 

Para.  7.1  Phase  A  Project 
Initiation 

Step  A4 

WBS  Template  1  Oct  2007 

ARDEC  101  Project 
Management  Procedure 

PIO 

Project  Planning  Technical 
Planning. 

Earned  Value 
Management  (EVM) 

Tool 

APO 

To  better  ensure  the  total  integration 
of  cost,  schedule,  and  work  scope 
aspects  of  the  contract. 

PIO 

Project  Planning.  Technical 
Planning.  Risk 

Management, 

Project  Plan 
Evaluation  Tool 

Too! 

ALL 

To  better  evaluate  the  quality  of  the 
Project  Plan 

Project  Plan  Evaluation 
Tool  21  Sept  2007 

PIO 

SED 

Project  Planning.  Technical 
Planning. 

Report  provides  tailored  list  of  Resources 


Verification  Tool 


□  Use  Interview 

□  Use  Questionnaires 

□  Include  Stakeholders  Early  and  Often. 

□  Have  Stakeholders  Peer  Review  Requirements 

□  Use  a  JCCB 


O  Microsoft  Access  -  [Ammo  Handling  Verification  Questionnaire  Form] 


m  File  Edit  Insert  Records  Window  Help 


-  0 


Paragraph: 

Interface  Definition 

TRL: 

r  TRL5 

IPT  Name:  Ammo  Handling 

Section: 

3. 1.2. 0-1 

F  TRLG 
r  TRL7 

Module  Name  AHR 

Requirements: 

The  Ammo  Handling  Subsystem  will  interface  with  the  T urret  Structure,  Gun 
Assembly,  Fire  Control,  Ammo  Suite  and  Secondary  Armament. 

ATD/Objective  Force: 

This  enables  and  disables 
the  required  field  warning: 

Switch  to  View  Mode  | 

TRL  5  Verification  Method: 

Responsibility: 

Location  of  Verif: 

Verification  Procedure:  Briefly 
If  the  requirement  will  not  be  v 


Data  Collected: 


Analysis 

Inspection 

Measurement 

Test 

N/A 


"Please  Select  a  Method" 


(e.g.:  IPT  Name,  Subcontractor,  System  Integrartor)  Critical  Test: 

(e.g.:  Picatimy,  Contractor  Facility.  Proving  Ground)  |~~Please  Select  a  Test~~ 
d  at  this  TRL  level  to  validate  or  confirm  the  requirement. 


TRL  G 


Verification  Method: 
Responsibility: 
Location  of  Verif: 


j  Please  Select  a  Method  3] 

I - 


(e.g.:  IPT  Name,  Subcontractor,  System  Integrartor) 
(e.g.:  Picatinny,  Contractor  Facility,  Proving  Ground) 


Verification  Procedure:  Briefly  describe  the  procedure  you  recommend  at  this  TRL  level  to  validate  or  confirm  the  requirement. 
If  the  requirement  will  not  be  verified  at  this  time  please  indicate  so: 


Critical  T  est: 

j  Please  Select  a  T est  T1 


Data  Collected: 


Clear 


Reset 


TRI  7  Verification  Method- I  Please  Select  a  Method  T  I 


1  < 

<- 

>  |  >1 

Record  | 

of  | 

Exit 

1 

1  1 

Templates 


□ 

isting  of  some  Templates: 

■  Project  Plan  Template 

■  Requirements  Management  Plan  Template 

■  System  Specification  Template 

■  Interface  Control  Document  Template 


Substantiating  Data 


□ 

his  year  we  are  working  on  metrics  and 
measures  that  will  provide  greater  insights  into 
what  is  and  isn’t  working 

□ 

here  is  a  whole  suite  of  metrics  and 
measurement  tools  that  have  been  developed. 


What  makes  a  “good”  tool? 


□ 

onfiguration  Management  built  into  the  tool  for 
Change  History,  versioning  etc. 

□ 

nstructions  on  how  to  use  the  tool 

■  Instruction  sheet,  pop-up  comments 

□ 

rocess  Flow 

□ 

eedback  Form 

a 

ie  the  tool  into  the  process  they  are  seeking  to 
implement  (language,  steps  etc.) 


Conclusion 


□ 

ools  are  common  focal  points  for  discussion. 

□ 

anagement  expects  them  to  be  used 

□ 

e  are  starting  to  capture  metrics  to  help  guide 
future  changes  and  to  build  a  case  to  develop 
and  make  improvements  to  tools. 


Questions? 


Educating  the  Next 
Generation  of  Software 
Engineers 


Art  Pyster 

Distinguished  Research  Professor 
Stevens  Institute  of  Technology  and 
Deputy  Executive  Director,  SER-UARC 


STEVENS 


Institute  of' Technology 


October  22,  2008 
art.pyster@stevens.edu 


lift  ASysT 

INSTITUTE 


Agenda 


•  How  the  world  has  changed 

•  The  current  state  of  software  engineering  education 

•  A  new  reference  curriculum 
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A  History  Lesson  -  1996 


Resolved:  Software  Should  Lead  in  Systems 
Engineering 

Jim  Armstrong  vs.  Art  Pyster 

The  systems  engineering  community  has  long  debated  the  extent  to 
which  software  disciplines,  processes,  and  practitioners  should 
influence  systems  engineering.  In  August  1996,  the  authors  held  a 
lively  debate  at  a  meeting  of  the  Washington  Metropolitan  Area 
Chapter  of  INCOSE  on  the  proper  role  of  software  engineering  within 
systems  engineering.  The  particular  issue  debated  was  the  proposal 
that  software  ideas,  process,  and  people  should  be  in  the  lead  when 
building  complex  systems.  Pyster  favored  that  view  while  Armstrong 
opposed  it. 


School  of 

Sy&Lrn&E.S  Enterprise 


Software  will  be  the  center  of 

systems  design. 


Eberhardt  Rechtin, 
1993 


Twenty  years  from  now,  software 
people  will  be  sitting  at  the  table 
and  the  other  disciplines  will  be 
sitting  around  the  sides  of  the 
room. 


Eberhardt  Rechtin, 
1993 
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What  do  we  teach  for  a  master’s 
degree  in  software  engineering? 


•  The  last  effort  to  create  a  reference  curriculum  for  graduate 
software  engineering  education  was  by  the  SEI  in  the  early 


1990s. 


•  There  are,  in  effect,  no  current  community-endorsed 
recommendations  on  what  to  teach  software  engineers  - 
nothing  that  recognizes  how  the  world  has  changed. 

•  Response:  create  a  project  to  create  a  new  reference  curriculum 
in  software  engineering 
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The  Integrated  Software  and 
Systems  Engineering  Curriculum  Project 


•  Begun  in  May  2007  at  Stevens  Institute  of  Technology 

•  Sponsored  by  DoD  Director  of  Systems  and  Software 
Engineering 

•  Three  products  planned: 

1.  A  modem  reference  curriculum  for  a  master’s  degree  in  software  engineering 
that  integrates  an  appropriate  amount  of  systems  engineering 

2.  A  modem  reference  curriculum  for  a  master’s  degree  in  systems  engineering 
that  integrates  an  appropriate  amount  of  software  engineering 

3.  A  tmly  interdisciplinary  degree  that  is  neither  systems  nor  software 
engineering  -  it  is  both 
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1st  Project  -  Graduate  Software 
Engineering  Reference  Curriculum 


1 .  Understand  the  current  state  of  SwE  graduate  education 
(November  2007) 

2.  Create  GSwERC  0.25  with  a  small  team,  suitable  for  limited 
review  (February  2008) 

3.  Publicize  effort  through  conferences,  papers,  website,  etc 
(continuous) 

4.  Obtain  endorsement  from  INCOSE,  NDIA,  ACM,  IEEE,  and 
other  professional  organizations  (continuous) 

5.  Create  GSwERC  0.50  suitable  for  broad  community  review 
and  early  adoption  (October  2008) 

6.  Create  GSwERC  1.0  suitable  for  broad  adoption  (2009) 
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Schoni  of 

Syslr msA  EnlcrpiiiiM 


The 

Rick  Adcock,  Cranfield  University  and  INCOSE 
Edward  Alef,  General  Motors 
Bruce  Amato,  Department  of  Defense 
Mark  Ardis,  Rochester  Institute  of  Technology 
Larry  Bernstein,  Stevens  Institute  of  Technology 
Barry  Boehm,  University  of  Southern  California 
Pierre  Bourque,  Quebec  University  and  SWEBOK 
volunteer 

John  Bracket,  Boston  University 
Murray  Cantor,  IBM 

Lillian  Cassel,  Villanova  and  ACM  volunteer 
Robert  Edson,  ANSER 

Richard  Lairley,  Colorado  Technical  University 
Dennis  Lrailey,  Raytheon  &  Southern  Methodist 
University 

Gary  Hafen,  Lockheed  Martin  and  NDIA 
Thomas  Hilbum,  Embry-Riddle  Aeronautical 
University 

Greg  Hislop,  Drexel  University  and  IEEE 
Computer  Society  participant 
Dave  Klappholz,  Stevens  Institute  of  Technology 
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evolving  author  team 

Philippe  Kruchten,  University  of  British  Columbia 
Phil  Laplante,  Pennsylvania  State  University,  Great 
Valley 

Qiaoyun  (Liz)  Li,  Wuhan  University,  China 
James  McDonald,  Monmouth  University 
John  McDermid,  University  of  York,  UK 
Ernest  McDuffie,  National  Coordination  Office  for 
NITRD 

Bret  Michael,  Naval  Postgraduate  School 
William  Milam,  Ford 

Ken  Nidiffer,  Software  Engineering  Institute 
Art  Pyster,  Stevens  Institute  of  Technology 
Doug  Schmidt,  Vanderbilt  University 
Mary  Shaw,  Carnegie  Mellon  University 
Robert  Suritis,  IBM 

Richard  Thayer,  California  State  University  at 
Sacramento 

Barrie  Thompson,  Sunderland  University,  UK 
Richard  Turner,  Stevens  Institute  of  Technology 
Joseph  Urban,  Texas  Technical  University 
Ricardo  Valerdi,  MIT  &  INCOSE 
David  Weiss,  Avaya 

Mary  Jane  Willshire,  Colorado  Technical  University 


Methodology  to  understand 
current  state  of  SwE  education 


•  Diverse  set  of  universities  with  Masters  programs  in  SWE 


Vary  in  size,  geography,  maturity,  resources,  target  market,  . . . 

Focused  on  programs  with  degree  in  SWE  or  Computer  Science  with  a  SWE 
specialization  -  not  degrees  in  information  technology  and  related  areas 


•  Used  Software  Engineering  Body  of  Knowledge  (SWEBOK) 
as  the  primary  framework  for  SWE  competencies 

•  Collected  data  from  school  websites 

Degree,  faculty  size,  student  population,  target  market,  . . . 

Degree  structure,  individual  course  descriptions 
Map  between  courses  and  SWEBOK 


Validated  data  with  one  or  more  professors  from  each  school 
Analyzed  for  commonalities  and  uniqueness 


1 .  Air  F orce  Institute  of  T echnology 

2.  Brandeis  University 

3.  California  State  University  -  Fullerton 

4.  California  State  University  - 
Sacramento 

5.  Carnegie  Mellon  University 

6.  Carnegie  Mellon  University  West 

7.  DePaul  University 

8.  Drexel  University 

9.  Dublin  City  University  (Ireland)  * 

10.  Embry-Riddle  Aeronautical  University 

1 1 .  George  Mason  University 

12.  James  Madison  University 

13.  Mercer  University 

14.  Monmouth  University 


Schools  studied 


15.  Naval  Postgraduate  School 

16.  Penn  State  University  -  Great  Valley 

17.  Quebec  University  (Canada)  * 

1 8 .  Rochester  Institute  of  T echnology 

19.  Seattle  University 

20.  Southern  Methodist  University 

2 1 .  Stevens  Institute  of  T echnology 

22.  Texas  Tech  University 

23.  University  of  Alabama  -  Huntsville 

24.  University  of  Maryland  University 
College 

25 .  University  of  Michigan  -  Dearborn 

26.  University  of  Southern  California 

27.  University  of  York  (UK)  * 

28.  Villanova  Univers%on-l/S  Schools 
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Schoiii  of 
Systems^  EnterpiiiFi 


Observations  from  28  schools 

1:  SWE  is  largely  viewed  as  a  specialization  of  Computer  Science  - 
much  as  systems  engineering  was  often  viewed  as  specialization  of 
industrial  engineering  or  operations  research  years  ago 

2.  Faculty  size  is  small  -  few  dedicated  SWE  professors,  making 
programs  relatively  brittle 

3.  Student  enrollments  are  generally  small  compared  to  CS  and  to 
other  engineering  disciplines 

4.  Many  programs  specialize  to  specific  markets  such  as  defense 
systems  or  safety  critical  systems 

5.  The  target  student  population  varies  widely  -  anyone  with 
Bachelors  and  B  average  to  someone  with  CS  degree  and  2+  years 
of  experience 


6.  Online  course  delivery  is  popular 
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More  observations 

software  developer  to  researcher  to 


Objective  for  graduates  vary  widely  - 
software  manager 


8.  Wide  variation  in  depth  and  breadth  of  SWEBOK  coverage  in  required 
and  semi-required*  courses 


9. 


Many  programs  have  required  or  semi-required  courses  that  cover 
material  that  is  either  not  in  the  SWEBOK  at  all  or  is  not  emphasized  in 


the  SWEBOK 


10.  Some  significant  topics  are  rarely  mentioned  -  agility,  software 
engineering  economics,  systems  engineering 

1 1 .  Some  topics  are  ubiquitous  -  formal  methods  and  architecture 

12.  “Object-oriented”  is  the  standard  development  paradigm  -  creating  a 
“clash”  with  many  systems  engineering  programs  that  emphasize 
structured  methods 


*A  student  has  a  50%  or  greater  probability  of  taking  a  semi-required  course. 
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'S 


■1 


Diverse  focuses 


1 .  Development  of  defense  systems 

2.  Acquisition  of  defense  systems 

3 .  Embedded  real-time  systems  - 

No  focus 

4.  Entrepreneurial  technology  companies  dominated 

5 .  Quantitative  software  engineering 

6.  Software  economics 

7.  Safety  critical  systems 

8.  Secure  software  engineering 

9.  Highly  dependable  software  systems 
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Entrance  requirements 


100 

90 

80 

70 

60 

50 

40 

30 

20 

10 

0 


Most  programs  offer  leveling 
courses  for  students  lacking 
entrance  requirements 


Many  programs  routinely  waive 
academic  requirements  for  students 
with  industrial  experience 


Know  how  to  Degree  in  Eng/Sci  /  Degree  in  Computing  Industry  Experience 
Program  Math 
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Sch  on  1  nJ 

Ky&Lrms.S  Enterprise 


required  and 


SWEBOK  coverage  in 
semi-required  courses 


The  approach  -  GSwERC  0.50 


1 .  Understand  the  current  state  of  S WE  graduate  education 
(November  2007) 

2.  Create  GSwERC  0.25  with  a  small  team,  suitable  for  limited 
review  (February  2008) 

3.  Publicize  effort  through  conferences,  papers,  website,  etc. 

(continuous) 

4.  Obtain  endorsement  from  ACM,  IEEE,  INCOSE,  NDIA,  and 
other  professional  organizations  (continuous) 

5.  Create  GSwERC  0.50  suitable  for  broad  community  review 
and  early  adoption  (October  2008) 

6.  Create  GSwERC  1.0  suitable  for  broad  adoption  (2009) 
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1 .  The  equivalent  of  an  undergraduate  degree  in  computing  or 
an  undergraduate  degree  in  an  engineering  or  scientific  field 
and  a  minor  in  computing 


2.  The  equivalent  of  an  introductory  course  in  software 
engineering 

3 .  At  least  two  years  of  practical  experience  in  some  aspect  of 
software  engineering  or  software  development. 
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Outcomes  1  to  4  at  graduation 


1 .  Mastered  the  Core  Body  of  Knowledge 


2.  Mastered  at  least  one  application  domain,  such  as  finance,  medical, 
transportation,  or  telecommunications,  and  one  application  type, 
such  as  real-time,  embedded,  safety-critical,  or  highly  distributed 
systems.  That  mastery  includes  understanding  how  differences  in 
domain  and  type  manifest  themselves  in  both  the  software  itself 
and  in  their  engineering,  and  includes  understanding  how  to  learn  a 
new  application  domain  or  type. 


3.  Mastered  at  least  one  knowledge  area  or  sub-area  from  the  CBOK 
to  at  least  the  Bloom  Synthesis  level. 


4.  Demonstrated  how  to  make  ethical  professional  decisions  and 
practice  ethical  professional  behavior. 
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-'•j Outcomes  5  to  7  at  graduation 

5 .  Understand  the  relationship  between  software  engineering 
and  systems  engineering  and  be  able  to  apply  systems 
engineering  principles  and  practices  in  the  engineering  of 
software. 

6.  Be  able  to  work  effectively  as  part  of  a  team,  including  teams 
that  may  be  international  and  geographically  distributed,  to 
develop  quality  software  artifacts,  and  to  lead  in  one  area  of 
project  development,  such  as  project  management, 
requirements  analysis,  architecture,  construction,  or  quality 
assurance. 

7.  Show  ability  to  reconcile  conflicting  project  objectives, 
finding  acceptable  compromises  within  limitations  of  cost, 
time,  knowledge,  existing  systems,  and  organizations. 
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-'•j Outcomes  8  to  10  at  graduation 

8.  Understand  and  appreciate  the  importance  of  feasibility 
analysis,  negotiation,  effective  work  habits,  leadership,  and 
good  communication  with  stakeholders  in  a  typical  software 
development  environment. 

9.  Understand  how  to  learn  new  models,  techniques,  and 
technologies  as  they  emerge,  and  appreciate  the  necessity  of 
such  continuing  professional  development. 

10.  Be  able  to  analyze  a  current  significant  software  technology, 
articulate  its  strengths  and  weaknesses,  and  specify  and 
promote  improvements  or  extensions  to  that  technology. 
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Additional  material  in  GSwERC 


•  Comparison  of  existing  graduate  software  engineering 
programs  with  GSwERC  recommendations  -  know  how  big 
the  gap  is  between  recommendations  and  practice 

•  Strategies  recommended  by  the  authors  to  implement 
GSwERC 

•  Hypothetical  modifications  of  existing  programs  to  more  fully 
satisfy  GSwERC 
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Systems  A  Enterprises 


Reviewers,  authors, 


SEEKING  AUTHORS,  REVIEWERS 
AND  EARLY  ADOPTERS  FOR  VERSION  0.5 


A  New  Reference 
Curriculum  for  Graduate 
Studies  Leading  to  a 

Master’s  Degree  in 
Software  Engineering 


Sines  August  2D07,  a  group  of  mare  than  30  professionals  from  academia.  Indus¬ 
try  and  government  have  bean  daveloplng  a  new  reference  curriculum  leading  to  a 
Maxtor's  Degree  In  Software  Engineering.  The  new  reference  cumculum  will 
integrate  systems  engineering  Into  the  education  of  software  engineers  and  reflect 
the  dramatic  changes  In  how  software  is  used  and  developed  since  the  last  major 
graduate  reference  curriculum  was  published  In  lire  eery  1990's.  Tha  effort  Is 
endorsed  hy  the  International  Council  on  Systems  Engineering  MNOdSE),  and 
the  LIS.  National  Defense  Industrial  Association  (NDIA)  Systems  Engineering 
Division.  The  IEEE  Computer  Society  has  a  participating  author  and  the  ACM  has 
a  volunteer  contributor.  Sponsorship  and  Funding  for  this  effort  are  being  provided 
bylho  U.S.  Department  of  Defense. 

Veision  0.25  of  the  Graduate  Software  Engineering  Relerence  Curriculum 
(GSwERC)  was  released  In  February  2QQU  fora  limited  review.  Version  0,5  is  being 
readied  for  worfchtfde  release  at  the  end  of  October  21109.  The  document  will  be 
posted  on  the  GSwERC  website  (www.  GSwERC, org).  Review  comments  from  all 
Interested  professionals  are  being  sought  Version  1.G  baseline  Is  expected 
sometime  In  2M&. 

A  dear  limitation  of  tha  eltort  so  far  has  been  the  demographics  of  tha  author  team. 
While  the  team  Is  extraordinarily  lalented  and  dedicated,  the  authors  of  these  lira 
Wo  drafts  were  mainly  from  the  United  States.  The  First  round  of  reviewers  was 
more  International,  hut  still  dominated  by  professionals  from  thy  U.S,  In  order  fo 
ensure  that  GSwERC  has  gloha  I  applicability,  we  seek  In  further  broaden  both  Ihe 
reviewers  for  the  second  draft  and  the  authorteam  for  Version  1 .0, 

One  of  tha  novel  Features  of  GSwERC  is  the  inclusion  of  a*p|  tit  comparisons  of 
existing  graduate  software  engineering  programs  to  GSwERC  recommendations 
and  the  Inclusion  of  hypothetical  modifications  to  two  of  those  programs  fo  better 
match  GSwERC-  These  comparisons  and  modifications  offer  a  window  on  how  well 
GSwERC  aligns  with  existing  practice  and  will  help  laeulty  understand  how  to  adopt 
GSwERC  in  thelrawn  unlvarsitles.  We  welcome  additional  comparisons  and  hypo¬ 
thetical  modifications  f  rom  other  un  rverslties  to  provide  more  insight  Info  the  gap 
between  GSwERC  and  current  practice  and  howto  close  that  gap. 


The  Graduate  Software  Engineering  Relerence 
Curriculum  (GSwERC)  Is  ready  for  early  adop¬ 
tion.  Several  of  the  current  authors  are  new 
integrating  portions  of  GSwERC  in  their  own 
programs..  We  are  also  Interested  In  helping 
universities  begin  to  adopt  GSwERC  in  2(M 

Please  visit  our  website  at 

www.GSwERC.wg 

for  more  Information. 

Dr  Art  Pyeter,  Distinguished  Research 
Professor,  from  Stevens  institute  of 
Technology  in  the  United  Slates,  le  the 
project  leader  for  GSwERC. 

Please  contact  him  at 

4rLpysteb&5le¥iEft9,edl  If  you  are 

Interested  in  participating. 


STEVENS 

Tnarhurw  rf  TMiwjr^y 

<f|  ASysT' 

institute  ^ 


and  early  adopters 


www.GSwERC.org 
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Outline 


■  The  Systems  Quality  Challenge 

■  Architecture  And  Quality  Defined 

■  Quality  Attribute-Based  Approaches  To 
Architecting  Systems 

■  Making  The  Case  For  Architectural  Quality 

■  Customer  Implications  Of  Quality-Attribute- 
Based  Architectural  Approaches 

■  Process  Maturity  And  Product  Quality 

■  A  Current  Concern:  Architecting  For 
System  Assurance 

■  Summary 
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It’s  About  The  Architecture  .  .  . 


One  of  the  top  ten  emerging  systemic 
issues,  from  fifty-two  in-depth  program 
reviews  since  March  2004,  was 
inadequate  software  architectures 


Source:  D.  Castellano.  Systemic  Root  Cause  Analysis.  NDIA  Systems 
Engineering  Division  Strategic  Planning  Meeting,  December,  2007. 
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It’s  Also  About  Quality  .  . 


■  The  NDIA  Top  Software  Issues  Workshop  examined 
the  current  most  critical  issues  in  software 
engineering  that  impact  the  acquisition  and 
successful  deployment  of  software-intensive 
systems 

■  Two  issues  emerged  that  were  focused  specifically 
on  the  relationship  between  software  quality  and 
architecture: 

-  Ensure  defined  quality  attributes  . . .  are  addressed  in 
requirements,  architecture,  and  design. 

-  Define  software  assurance  quality  attributes  that  can 
be  addressed  during  architectural  trade-offs 


Source:  G.  Draper  (ed.),  Top  Software  Engineering  Issues  Within  Department  of  Defense 
and  Defense  Industry.  National  Defense  Industrial  Association,  Arlington,  VA,  August  2006. 
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The  Systems  Quality 
Challenge 


■  If  we  are  successful  in  managing  risk  for 
the  systems  we  build,  and  meet 
stakeholder  expectations,  we  must: 

-  Start  as  early  as  possible  in  the  design 
process  to  understand  the  extent  to  which 
those  expectations  might  be  achieved 

-  Develop  candidate  system  architectures  and 
perform  architecture  trade-offs 

-  Define  and  use  a  set  of  quantifiable  system 
attributes  tied  to  stakeholder  expectations, 
against  which  we  can  measure  success 
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The  Systems  Quality  Challenge  Is 
A  Software  Quality  Challenge 


■  Most  systems  we  encounter  today 
contain  software  elements  and  most 
depend  upon  those  software 
elements  for  a  good  portion  of  their 
functionality 

■  Modern  systems  architecture  issues 
cannot  be  adequately  addressed 
without  considering  the  implications 
of  software  architecture 
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Architecture  Defined 


■  The  fundamental  organization  of  a  system  embodied 
in  its  components,  their  relationships  to  each  other, 
and  to  the  environment,  and  the  principles  guiding  its 
design  and  evolution. 

Source:  IEEE  1471-2000,  IEEE  Recommended  Practice  for  Architectural 
Description  of  Software-Intensive  Systems.  The  Institute  of  Electrical  and 
Electronics  Engineers,  Inc.,  New  York,  NY,  2000. 

■  The  set  of  all  of  the  most  important,  pervasive, 
higher-level,  strategic  decisions,  inventions, 
engineering  trade-offs,  assumptions,  and  their 
associated  rationales  concerning  how  the  system 
meets  its  allocated  and  derived  product  and  process 
requirements 


Source:  D.  Firesmith,  P.  Capell,  D.  Falkenthal,  C.  Hammons,  D.  Latimer,  and  T.  Merendino.  The 
Method-Framework  for  Engineering  System  Architectures  (MFESA):  Generating  Effective  and 
Efficient  Project-Specific  System  Architecture  Engineering  Methods.  November,  2008.  CRC  Pr  I  Lie, 
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Quality  Defined 


■  Software  quality:  The  degree  to 
which  software  possesses  a  desired 
combination  of  attributes. 

Source:  IEEE  Standard  1061-1992.  Standard  for  a  Software  Quality  Metrics 
Methodology.  New  York:  Institute  of  Electrical  and  Electronics  Engineers,  1992. 


■  Software  product  quality:  The  totality 
of  characteristics  of  an  entity  that 
bear  on  its  ability  to  satisfy  stated 
and  implied  needs. 

Source:  ISO/IEC  9126-1:  Information  Technology  -  Software  product  quality  - 
Part  1:  Quality  model.  ISO,  Geneva  Switzerland,  2001. 
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Quality  Attribute-Based  Approaches 
To  Architecting  Systems 


Developing  systematic  ways  to  relate  the 
software  quality  attributes  of  a  system  to 
the  system’s  architecture  provides  a  sound 
basis  for  making  objective  decisions  about 
design  tradeoffs  and  enables  engineers  to 
make  reasonably  accurate  predictions 
about  a  system’s  attributes  that  are  free 
from  bias  and  hidden  assumptions.  The 
ultimate  goal  is  the  ability  to  quantitatively 
evaluate  and  trade  off  multiple  software 
quality  attributes  to  arrive  at  a  better 

overall  system.  Source:  M.  Barbacci,  M.  Klein,  T.  Longstaff,  and  C.  Weinstock. 

Quality  Attributes,  CMU/SEI-95-TR-021 .  Software  Engineering 

Institute,  Carnegie  Mellon  University,  December  1 995. 
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Relationships  Between 

Attributes 


■  Collaboration 

-  Increasing  the  degree  to  which  one  attribute 
is  realized  increases  the  realization  of 
another 

■  Damage 

-  Increasing  the  degree  to  which  one  attribute 
is  realized  decreases  the  realization  of 
another 

■  Dependency 

-  The  degree  to  which  one  attribute  is  realized, 
is  dependent  upon  the  realization  of  at  least 
some  sub-characteristics  of  another 


Source:  X.  Franch  and  J.  Carvallo.  “Using  Quality  Models  in 
Software  Package  Selection”,  IEEE  Software,  pp.  34-41.  New 
York:  Institute  of  Electrical  and  Electronics  Engineers,  2003. 
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Optimization  Among  Quality 

Attributes 


■  Example:  A  large  telecommunication 
application 

-  Good  optimization  (Collaboration) 

■  balance  among  multiple  quality  attributes,  such  as 
maintainability,  performance  and  availability 

-  Poor  optimization  (Damage) 

■  Focusing  solely  on  maintainability  often  results  in 
poor  system  performance 

■  Focusing  on  performance  and  availability  alone  may 
result  in  result  in  poor  maintainability 

■  Explicit  architectural  decisions  can  facilitate 
optimization  among  quality  attributes 

Source:  D.  Haggander,  L.  Lundberg,  and  J.  Matton,  “Quality  Attribute  Conflicts  -  Experiences  from  a  Large 
Telecommunication  Application,  ’’  Proceedings  of  the  Seventh  International  Conference  on  Engineering  of 
Complex  Computer  Systems  (ICECCS’01),  New  York:  Institute  of  Electrical  and  Electronics  Engineers,  2001. 
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Understanding  Quality  In  The  Context 
Of  Architectural  Structures 


■  Structures  for  describing  architectures 

-  Functional  structure  is  the  decomposition  of  the  functionality  that  the 
system  needs  to  support 

-  Code  structure  is  the  code  abstractions  from  which  the  system  is 
built 

-  Concurrency  structure  is  the  representation  of  logical  concurrency 
among  the  components  of  the  system 

-  Physical  structure  is  just  that,  the  structure  of  the  physical 
components  of  the  system 

-  Developmental  structure  is  the  structure  of  the  files  and  the 
directories  identifying  the  system  configuration  as  the  system 
evolves 

■  Using  architectural  structures  to  understand  quality 

-  Concurrency  and  Physical  structures  are  useful  in  understanding 
system  Performance 

-  Concurrency  and  Code  structures  are  useful  in  understanding 
system  Security 

-  Functional,  Code,  and  Developmental  structures  are  useful  in 
understanding  system  Maintainability 

Source:  L.  Bass  and  R.  Kazman,  Architecture-Based 
Development,  CMU/SEI-99-TR-007.  Software  Engineering 
Institute,  Carnegie  Mellon  University,  April  1 999. 
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Attribute-Driven  Design 


■  Attribute-Driven  Design  (ADD)  produces  an  initial  software 
architecture  description  from  a  set  of  design  decisions  that 
show: 

-  Partitioning  of  the  system  into  major  computational  and 
developmental  elements 

-  What  elements  will  be  part  of  the  different  system  structures, 
their  type,  and  the  properties  and  structural  relations  they 
possess 

-  What  interactions  will  occur  among  elements,  the  properties  of 
those  interactions,  and  the  mechanisms  by  which  they  take 
place 

■  In  the  very  first  step  in  ADD,  quality  attributes  requirements 
are  expressed  as  the  system’s  desired  measurable  quality 
attribute  response  to  a  specific  stimulus 

■  Knowing  these  requirements  for  each  quality  attribute 
supports  the  selection  of  design  patterns  and  tactics  to 
achieve  those  requirements 

Source:  R.  Wojcik,  F.  Bachmann,  L.  Bass,  P.  Clements,  P.  Merson,  R.  Nord,  and  B. 

Wood,  Attribute-Driven  Design  (ADD),  Version  2.0,  CMU/SEI-2006-TR-023. 

Software  Engineering  Institute,  Carnegie  Mellon  University,  November  2006. 
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Understanding  The  Consequences  Of  Architectural 
Decisions  With  Respect  To  Quality  Attributes 


■  The  Architecture  Tradeoff  Analysis  MethodSM  (ATAMSM)  is 
dependent  upon  quality  attribute  characterizations,  like 
those  produced  through  ADD,  that  provide  the  following 
information  about  each  attribute: 

-  The  stimuli  to  which  the  architecture  must  respond 

-  How  the  quality  attribute  will  be  measured  or  observed  to 
determine  how  well  it  has  been  achieved 

-  The  key  architectural  decisions  that  impact  achieving  the 
attribute  requirement 

■  ATAM  takes  proposed  architectural  approaches  and 
analyzes  them  based  on  quality  attributes 

-  generally  specified  in  terms  of  scenarios  addressing  stimuli 
and  responses 

■  Use  case  scenarios,  describing  typical  uses  of  the  system 

■  Growth  scenarios,  addressing  planned  changes  to  the  system 

b  Exploratory  scenarios,  addressing  any  possible  extreme 
changes  that  would  stress  the  system 

■  ATAM  also  identifies  sensitivity  points  and  tradeoff  points 

Source:  R.  Kazman,  M.  Klein,  and  P.  Clements,  ATAM:  Method  for  Architecture  Evaluation, 
CMU/SEI-2000-TR-004,  Software  Engineering  Institute,  Carnegie  Mellon  University,  August  2000. 
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Some  Real  World  Architecture 

Review  Issues 


■  Results  from  four  AT&T  companies 
Between  1989  and  2000 

■  More  than  1,000  issues 

■  Six  classes  of  issues 

-  Product  architecture  and  design,  29-49% 

-  Management  controls,  14-26% 

-  Problem  definition,  1 0—1 8% 

-  Process,  4-19% 

-  Technology,  3-14% 

-  Domain  knowledge,  2-5% 

Source:  J.  Maranzano,  S.  Rozsypal,  G.  Zimmerman,  G.  Warnken,  P.  Wirth,  and  D.  Weiss, 
Architecture  Reviews:  Practice  and  Experience,  IEEE  Software,  March/April  2005. 
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Making  The  Case  For 
Architectural  Quality 


■  The  Quality  Case 

-  The  set  of  claims,  supporting  arguments,  and 
supporting  evidence  that  provide  confidence  that  the 
system  will  in  fact  demonstrate  its  expected  quality 
characteristics 

-  Common  types  of  quality  cases  include: 

■  safety  cases 

■  security  cases 

■  assurance  cases 

■  The  Architectural  Quality  Case 

-  The  architectural  claims,  supporting  arguments, 
including  architectural  decisions  and  tradeoffs, 
architectural  representations,  and  demonstrations  that 
the  architecture  will  exhibit  its  expected  quality 
characteristics 

Source:  D.  Firesmith,  P.  Capell,  D.  Falkenthal,  C.  Flammons,  D.  Latimer,  and  T.  Merendino.  The 
Method-Framework  for  Engineering  System  Architectures  (MFESA):  Generating  Effective  and 
Efficient  Project-Specific  System  Architecture  Engineering  Methods.  November,  2008.  CRC  Pr  I  Lie, 
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Risk  Management  Implications  Of  Quality- 
Attribute-Based  Architectural  Approaches 


■  Stakeholder  quality  requirements  will  have  been 
distilled  into  architectural  drivers  which  will  have 
shaped  the  system  architecture 

■  Tradeoffs  will  have  been  made  to  optimize  the 
realization  of  important  quality  characteristics,  in 
concert  with  stakeholder  expectations 

■  The  level  of  confidence  that  the  resultant 
architecture  will  meet  those  expectations  will  be 
known 

■  Stakeholders  will  be  knowledgeable  of  any  residual 
risk  they  are  accepting  by  accepting  the  delivered 
system 


Source:  R.  Wojcik,  F.  Bachmann,  L.  Bass,  P.  Clements,  P.  Merson,  R.  Nord,  and  B. 
Wood,  Attribute-Driven  Design  (ADD),  Version  2.0,  CMU/SEI-2006-TR-023. 
Software  Engineering  Institute,  Carnegie  Mellon  University,  November  2006. 
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Process  Maturity  Does  Not 
Guarantee  Product  Quality 


■  The  CMMI®  embodies  the  process 
management  premise  that,  the  quality 
of  a  system  or  product  is  highly 
influenced  by  the  quality  of  the  process 
used  to  develop  and  maintain  it 


■  However: 


Source:  CMMI®  for  Development,  Version  1.2, 
CMU/SEI-2006-TR-008,  Software  Engineering  Institute, 
Carnegie  Mellon  University,  August  2006 


Several  recent  program  failures  from 
organizations  claiming  high  maturity 
levels  have  caused  some  to  doubt 
whether  CMMI  ®  improves  the  chances  of 
a  successful  project 


Source:  R.  Hefner.  CMMI  Horror  Stories:  When  Good 
Projects  Go  Bad.  SEPG  Conference,  March  2006 
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.  .  .  But  Engineering  Discipline 

Might 


■  Process  maturity  can  in  many  cases 
improve  project  performance,  but 
special  attention  to  the  engineering 
processes  is  required  to  ensure  that 
stakeholder  quality  expectations  are 
realized  in  resultant  products. 
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A  Current  Concern:  Architecting 
For  System  Assurance 


■  The  challenge: 

Integrating  a  heterogeneous  set  of  globally  engineered 
and  supplied  proprietary,  open-source,  and  other 
software;  hardware;  and  firmware;  as  well  as  legacy 
systems;  to  create  well-engineered  integrated, 
interoperable,  and  extendable  systems  whose 
security,  safety,  and  other  risks  are  acceptable  -  or  at 
least  tolerable. 

Source:  P.  Croll,  “Engineering  for  System  Assurance  -  A  State  of  the  Practice 

Report,”  Proceedings  of  the  1st  Annual  IEEE  Systems  Conference.  New  York: 

Institute  of  Electrical  and  Electronics  Engineers,  April  2007 

■  The  vision: 


The  requirements  for  assurance  are  allocated  among 
the  right  systems  and  their  critical  components,  and 
such  systems  are  designed  and  sustained  at  a  known 
level  of  assurance. 

Source:  K.  Baldwin.  DOD  Software  Engineering  and 
System  Assurance  New  Organization  -  New  Vision, 

DHS/DOD  Software  Assurance  Forum,  March  8,  2007 
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Architectural  Principles  For 

Assurance 


■  Isolate  critical  components  from  less-critical 
components 

■  Make  critical  components  easier  to  assure  by 
making  them  smaller  and  less  complex 

■  Separate  data  and  limit  data  and  control  flows 

■  Include  defensive  components  whose  job  is  to 
Drotect  other  components  from  each  other  and/or 
:he  surrounding  environment 

■  Understanding  the  interrelationships  between 
components  and  their  linkages 

■  Use  defense-in-depth  measures  where  appropriate 

■  Beware  of  maximizing  performance  to  the  detriment 
of  assurance 


Source:  Engineering  For  System  Assurance,  Version  1.0.  National  Defense 
Industrial  Association,  System  Assurance  Committee,  Arlington,  Virginia 
October  2008. 
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Summary 


■  If  we  are  to  be  successful  in  managing 
risk  for  the  systems  we  build,  and  meet 
stakeholder  expectations,  we  must: 

-  Start  as  early  as  possible  in  the  design 
Drocess  to  understand  the  extent  to  which 
:hose  expectations  might  be  achieved 

-  Define  a  set  of  quantifiable  quality  attributes 
tied  to  stakeholder  expectations,  against 
which  we  can  measure  success  and 
understand  the  residual  risk  stakeholders  are 
being  asked  to  accept 

-  Develop  candidate  system  architectures  and 
perform  architecture  trade-offs  using  those 
attributes 
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Modernization  AND  Stability/Counterinsurgency 


talking  about  irregular  or  asymmetric  warfare, 

in  favor  ol  institutionalizing  counterinsurgency  skills,  and  our  ability  to  conduct  stability  and  support 
operations. 

The  need  for  the  state  of  the  art  systems  -  particularly  longer  range  capabilities  -  will  never  go  away,  as 
we  strive  to  offset  the  countermeasures  being  developed  by  other  nations.  But  at  a  certain  point,  given 
the  types  of  situations  we  are  likely  to  face,  it  begs  the  question  whether  specialized,  often  relatively 
low-tech  equipment  for  stability  and  counterinsurgency  missions  is  also  needed. 

•  How  do  we  institutionalize  procurement  of  such  capabilities  -  and  the  ability  to  get  them  fielded  quickly? 

•  Why  do  we  have  to  go  outside  the  normal  bureaucratic  process  to  develop  counter-IED  technologies,  to  build 
MRAPs,  and  to  quickly  expand  our  ISR  capability?  In  short,  why  did  we  have  to  bypass  existing  institutions  and 
procedures  to  get  the  capabilities  we  need  to  protect  our  troops  and  pursue  the  wars  we  are  in? 

Our  conventional  modernization  programs  seek  a  99  percent  solution  in  years.  Stability  and 
counterinsurgency  missions  -  the  wars  we  are  in  -  require  75  percent  solutions  in  months. 

•  The  challenge  is  whether  in  our  bureaucracy  and  in  our  minds  these  two  different  paradigms  can  be  made  to 
coexist. 

•  The  issue  then  becomes  how  we  build  this  kind  of  innovative  thinking  and  flexibility  into  our  rigid  procurement 
processes  here  at  home.  The  key  is  to  make  sure  that  the  strategy  and  risk  assessment  drives  the 
procurement,  rather  than  the  other  way  around. 

I  believe  we  must  do  this.  The  two  models  can  -  and  do  -  coexist. 


Extracted  from  speech  delivered  by  Secretary  of  Defense  Robert  M.  Gates, 
National  Defense  University,  Washington,  D.  C.  September  29,  2008 

http://www.  defense! ink.  mil/speeches/speech.  aspx?speechid=1279 
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There  are  three  diverging  tempos 


'^Arms-iength’  or  ‘smart’  Defense  Companies 

await  Requirements  expressed  in 
-  Programmes. 

Requirement 


Competitive  advantage  is  gained 
by  aligning  the  Composite 
Capabilities  to  the  Demand. 


Software  Engineering  Institute  CarnegieMellon 


i  and  Appropriate  Competition  in  C4ISTAR  Transformation ,  Dr  Nicholas  Whittall  RUSI  2007 

The  Value  Stairs: 
changing  the  value  equation 
Philip  Boxer,  October  21st  2008 
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The  divergence  of  tempos  challenges  the 
supplier  to  support  Type  III  Agility 


Collaborative 


Supplier 

Alignment 


Competitive. 


A 


Type  2  Agility 


Centre-drj. 

Collabefra 


Directed 

mposition 


Traditional  or 
‘SmafT  Engineering 


Type  3  Agility 


Edge-driven 
Collaboration 

Design  for 
Flexibility 
(Supporting  SoS 
extensibility) 


lyDe  1  Agility  Type  1+ Agility 


Directed 

Composition 

Workarounds,  UORs, 
etc 


Anticipated 


Unanticipated 


The  C4ISTAR 
Sector  net- 
enabled  journey 


Contingency 

Planning 


Operational  Alignment 

Derived  from  The  Double  Challenge’,  in  Boxer,  P.J.  et  al.  (2008)  SoS  Navigator  2.0:  A  Context-Based  Approach  to  System-of-Systems  Challenges  (CMU/SEI- 
2008-TN-001).  Software  Engineering  Institute,  Carnegie  Mellon  University,  2008.  http://www.sei.cmu.edu/publications/documents/08.reports/08tn001.html 
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Agenda 


The  demand  for  agility 
s>  Managing  alignment 
Creating  value  for  the  defense  enterprise 
Changing  the  value  equation 


Software  Engineering  Institute 


The  Value  Stairs: 

•  m  11  changing  the  value  equation 

Carnegie  Mellon  Philip  Boxer,  October  21st  2008 

©  2008  Carnegie  Mellon  University 


6 


The  approach  to  alignment  is  ‘stratified’ 


underlying 

technologies 


Capability  Alignment 
(to  Alignment  Tempo)  ( 


ultimate 

effects 


This  alignment  is 
‘Stratified’ 


Software  Engineering  Institute 


The  Value  Stairs: 

•  m  11  changing  the  value  equation 

Carnegie  Mellon  Philip  Boxer,  October  21st  2008 

©  2008  Carnegie  Mellon  University 


7 


The  divergence  of  these  tempos  creates  new 
challenges  for  the  Defense  Enterprise 


Supplier 
Alignment 
(to  Acquisition  Tempo) 

A 


Collaborative 

Supplier 

Alignment 

Competitive 


Type  2  Agility 

Type  3  Agility 

Centre-driven 

Collaboration 

Edge-driven 

Collaboration 

Design  for 
Integration 

Design  for 
Flexibility 
(Supporting  SoS 
extensibility) 

Type  1  Agility 

Type  1  + Agility 

Directed 

Directed 

Composition 

Composition 

Traditional  or 

Workarounds,  UORs, 

‘Smart’  Engineering 

etc 

Anticipated  Unanticipated 

Operational  Alignment 


What  kinds  of  agility  are 
needed  from  suppliers? 


Operational 


Alignment 

(to  Campaign  Tempo) 


Capability 
Alignment 
(to  Alignment  Tempo) 


What  kinds  of  value 
equation  are  involved 
here? 
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Capability 
Alignment 
(to  Alignment 


The  WHAT: 

Equipment, 
Platforms  etc 


The  HOW: 

Tier  1  Primes  and 
Unified  Customer 


The  WHO/M: 

Mission  Command 
of  Force  Structure 


The  WHY: 

Decisive 

Points 


5 


t 


Tempo) 


SCOPE 

(Competitive  context) 
Planning 


COLLABORATIVE 

MODEL 

(Collaboration) 

Governance 


BUSINESS 

MODEL 

(Conceptual) 

Owning 


EVENT 

(WHAT) 

e.g.  things  done 


DATA 

(WHAT) 
e.g.  data 


FUNCTION 

(HOW) 
e.g.  function 


NETWORK 

(WHERE) 
e.g.  network 


PEOPLE 

(WHO) 

e.g.  organisation 


TIME 

(WHEN) 
e.g.  schedule 


list of  Things  Important  to  the 
Business 


Entity  =  Class  of 
Business  Thing 


List  of  Processes-  the 
Business  Performs 


Representation  of 
the  physical 


alii 


e.g..  Semantic  Model 


i) 


Entity  =  Business  Entity 
Relationship  =  Business 
Relationship 


Process  =  Class  of 
Business  Process 


List  of  Locations  in  Which 
the  Business  Operates 


Node  =  Major  Business 
Location 


List  of  Organizations 
Important  to  the  Business 


People  -  Major 
Organizational  Unit 


List  of  Events.'Cvdes 
Significant  lo  the  Business 


Time  =  Major  Business 
Event/Cycle 


e.g^  Business  Process  Model 

Jt 

Process  =  Business  Process 
I/O  =  Business  Resources 


Multiple 

Enterprises 


e.g.  Business  Logistic  System 


Node  =  Business  Location 
Link  =  Business  Linkage 


e.g.,  Work  Flow  Model 


§i:- 

People  =  Organization  Unit 
Work  =  Work  Product 


USE  CONTEXT 

(WHO  for  WHOM) 
e.g.  particular  client 


MOTIVATION 

(WHY) 

e.g.  strategy 


Ends/Means  =  Major 
Business  Goal/Slrmlegy 


Cohesion  in 
relation  to 
demand/threat 
situation 


e.g.,  Master  Schedule 

EH 


Time  =  Business  Event 
Cycle  =  Business  Cycle 


e.g.,  Business  Plan 


End  =  Business  Objective 
Means  =  Business  Strategy 


SYSTEM 

MODEL 

(Logical) 

Designing 


TECHNOLOGY 

MODEL 

(Physical) 

Building 


e.g.,  Distributed  System 
Architecture 

I 

Node  =  I/S  Function  | 
(Processor,  Storage,  etc) 

Link  =  Line  Characteristics 


e.g.,  Business  Rule  Model 


DETAILED 

REPRESENTATION 

S 

(out-of -modeling-context) 
Subcontracting 


e.g..  Data  Definition 


e.g.,  Program 


Entity  =  Field  Process  =  Language  Statement 

Relationship  =  Address  I/O  =  Control  Block 


Source  of  coloured  squares:  Zachman  Framework,  www.zifa.com 


e.g.,  Network  Architecture 


Node  =  Address 
Link  =  Protocol 


.,  Security  Architecture 


People  =  Identity 
Work  =  Job 


Time  =  Interrupt 
Cycle  =  Machine  Cycle 


End  =  Sub-condition 
Means  =  Step 
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The  divergence  of  these  tempos  creates  new 
challenges*  for  the  Defense  Enterprise 

Supplier 

Alignment 


(to  Acquisition  Tempo) 

A 


Collaborative 


Supplier 

Alignment 


Type  2  Agility 

Type  3  Agility 

Centre-driven 

Collaboration 

Edge-driven 

Collaboration 

Design  for 
Integration 

Design  for 
Flexibility 
(Supporting  SoS 
extensibility) 

Type  1  Agility 

Type  1  + Agility 

Directed 

Directed 

Composition 

Composition 

Traditional  or 

Workarounds,  UORs, 

‘Smart’  Engineering 

etc 

How  does  the  role  of 
the  supplier  fit  in? 

Capability 
Alignment 
(to  Alignment  Tempo) 


What  kinds  of  agility  are 
needed  from  suppliers? 


Operational 
Alignment 
(to  Campaign  Tempo) 


How  does  the  DoD  generate  the  requisite 
variety  of  operational  behaviours? 


*  For  more  on  these,  see  Boxer,  P.J.  (2008)  SoS  Navigator  Principles  for  Sustaining  Dynamic  Alignment:  The  Example  of  U.  S.  Army  Acquisition  Strategies  and 
Operational  Realities,  Special  Report,  Software  Engineering  Institute,  Carnegie  Mellon  University,  CMU/SEI-2008-SR-027,  September  2008 
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Value  for  Defense  comes  from  managing  a 

Double  ‘V’  Military  Effects 


demand-side 


supply-side 


Capability  gap 

minus  DOTMLPF  = 

Requirement 


decomposition 


System 

integration 


System  components 


Boxer,  P.J.  (2007)  Managing  the  SoS  Value  Cycle,  January  2007,  http://www.asymmetricdesign.com/archives/85 


i 
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This  double  ‘V’  is  layered,  spanning  the  three 
different  kinds  of  tempo 


The  WHY 


The  WHO 
(in  relation  to) 
WHOM 

demand-side 
supply-side 

The  HOW 


Effects 


Decisive  Points  6  <V 

Joint  Command  O  Cx 

Composite  Capabilities ] 

Agile  Force  Structure  4 

Force  Elements 


Force  Element  3  Cx 

Fielded  Equipment  J 

Fielded  Equipment  2 

Equipment 


The  WHAT 


Equipment  1 


■  -  /  Zampaig  ] 

I  S V  V  Tempo  J) 


Pragmatic 

constraints 


The  nature  of  this  overlap 
depends  on  the  engineering 
constraints  being  under- 
-1 determinin p*- 


Engineering 

constraints 


*  Boxer,  P.J.  (2008)  Framework  Architectures,  Navigator  White  Paper,  Software  Engineering  Institute,  Carnegie  Mellon  University,  June  2008 
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These  contexts-of-use  have  to  be  related  to  the 
individual  capabilities 


Effects  Ladder 


X 


Many-to-many 

composition 


Variety  of  Scenarios 


Demand  for  Synchronization 


Demand  for  Composite  Capabilities 


demand-side 

supply-side 


Supply  of  Force  Elements 


Orchestration  of 
geommrids-of-use 


Effects 
6:  Decisive  Points 


j  Synchronization 

5:  Joint 
Command 


^Composite  Capabilities 


Pragmatic 

constraints 


4:  Agile  Force 
Structure 


) 


Force  Elements 


Boxer,  P.J.  et  al  (2008)  “  Systems-of-Systems  Engineering  and  the  Pragmatics  of  Demand,"  Proceedings  of  the  Second  Annual  IEEE  Systems 
Conference  pp107.  Montreal,  Quebec,  Canada,  April  7-10,  2008.  IEEE,  2008. 

http://www.  ieeexplore.  ieee.org/xpl/freeabs_all.  jsp?isnumber=451 8971  &arnumber=4519030&count=89&index=58 
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Adding  the  socio-technical  perspective  in 
relation  to  demand  extends  the  analytical  space 


Socio-technical  SoS  = 

SoS  foundation  + 
Organization  + 
Synchronization  views 


SoS  foundation  = 

Equipment  and 
Information  views 


The  analysis  puts  the 

Socio-technical  SoS 

in  relation  to  an 
Effects  View 


This  leads  to  a  different  kind  of  analysis  of 
interoperability... 


Socio-technical  SoS  in 
relation  to  Demand 


Distinguishing  three 
different  kinds  of  path 


Functional  Demand 

Coupling  cohesion 


▼ 


Accountability 

Hierarchies 


Analyzing  alignment 


of  strata 


mission  problem 


constituent 

capabilities 


Identifying  Interoperability  Gaps 
in  the  different  strata 


demand 

situations 


Source:  Anderson,  Boxer  &  Browsword  (2006)  An  Examination  of  a  Structural  Modeling  Risk  Probe  Technique,  Special  Report,  Software  Engineering 
Institute,  Carnegie  Mellon  University,  CMU/SEI-2006-SR-017,  October  2006.  http://www.sei.cnnu.edu/publications/docunrients/06.reports/06sr017.htnnl 

Special  permission  to  use  PAN  in  this  Technical  Probe  was  granted  by  Boxer  Research  Limited. 
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Spanning  the  layers  means  managing  different 
kinds  of  value  equations 


demand-side 

supply-side 


Decisive  Points 

6 

Mission  Command 

5 

Agile  Force  Structure 

4 

Force  Element 

3 

Fielded  Equipment 

2 

Equipment 

1 

Type  3 


Type  2 


Type  1 


■  ■  /  fCampaigm  j 

|  IV  Tempo  I) 


Military  Command  +  Unified  Customer  +  Supplier 


r 

Unified  Customer  +  Supplier 


Supplier 


The  Value  Stairs: 
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The  Value  Stairs:  a  progressive  development  of  the 
value  equation  model 


Type  I 


I  I  ‘Above  the  customer  strategy  ceiling’ 


Through 

Life-cycle 

(of  equipment 
or  platform) 

_ _ i _ , 

i  i 

Arms-  ‘Smart’ 
length 


|  Purchaser  pre-contractual 
!  |  Provider  Contract 

I  I  Provider  subcontracted 


Decisive  Points 

6 

Mission  Command 

5 

Agile  Force  Structure 

4 

Force  Element 

3 

- 1 

Fielded  Equipment 

2 

Equipment 

1 

►  ‘strategy  ceiling’ 

The  purchaser  is  buying; 


Defense  Equipment  &  Support 
Defense  Equipment 
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The  Value  Stairs:  a  progressive  development  of  the 
value  equation  model 


Decisive  Points 

Mission  Command 
Agile  Force  Structure 
Force  Element 
Fielded  Equipment 
Equipment 


Type  I 

Through 

Life-cycle 
(of  equipment 
or  platform) 

i , 

Arms-  ‘Smart’ 
length 


Type  II 

Through-Life 

(equipment-  or 
platform-based) 

Capability 

r - 1 - 1 

TLAM  TLCM 

(availability  (capability 


I  I  ‘Above  the  customer  strategy  ceiling’ 
|  Purchaser  pre-contractual 
!  |  Provider  Contract 

I  I  Provider  subcontracted 
I  I  Potential  for  open-sourcing 

►  ‘strategy  ceiling’ 

The  purchaser  is  buying; 


Operational  Military  Capability 
Military  Capability  across  DOTMLPF 
Defense  Equipment  &  Support 
Defense  Equipment 
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The  Value  Stairs:  a  progressive  development  of  the 
value  equation  model 


The  ‘plus’  in 
TLCM+  indicates 
that  what  is  being 
supplied  is  a  SoS 
the  definition  of 
which  is  not 
equipment-b  ased 
or  platform-based 


Decisive  Points 


Mission  Command 
Agile  Force  Structure 
Force  Element 
Fielded  Equipment 
Equipment 


Type  ! 

Through 

Life-cycle 

(of  equipment 
or  platform) 


i  i 

Arms-  ‘Smart’ 


Type  III 

Through-Life  Through-Life 
Composite 
Capability 

Capability 


JL 


length 


li 


TLAM  TLCM 

(availability  (capability 
management)  management) 


| _ |  ‘Above  the  customer  strategy  ceiling’ 

|  Purchaser  pre-contractual 
[j  Provider  Contract 
I  I  Provider  subcontracted 
I  I  Potential  for  open-sourcing 


TLCM+ 


►  ‘strategy  ceiling’ 


The  purchaser  is  buying: 
Operational  Mission  Capability 

Operational  Military  Capability 

Military  Capability  across  DOTMLPF 

Defense  Equipment  &  Support 

Defense  Equipment 
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The  value  equation  must  evolve  as  the  demand 
for  the  variety  of  operational  behaviors  changes 

Supplier 

Alignment 


(to  Acquisition  Tempo) 

A 


Collaborative 


Supplier 

Alignment 


Type  2  Agility 

Type  3  Agility 

Centre-driven 

Collaboration 

Edge-driven 

Collaboration 

Design  for 
Integration 

Design  for 
Flexibility 
(Supporting  SoS 
extensibility) 

Type  1  Agility 

Type  1  + Agility 

Directed 

Directed 

Composition 

Composition 

Traditional  or 

Workarounds,  UORs, 

‘Smart’  Engineering 

etc 

How  does  the  role  of 
the  supplier  fit  in? 

Capability 
Alignment 
(to  Alignment  Tempo) 


What  kinds  of  agility  are 
needed  from  suppliers? 


Operational 
Alignment 
(to  Campaign  Tempo) 


^7  ^17  t'jr 


How  does  the  DoD  generate  the  requisite 
variety  of  operational  behaviours? 


The  value  equation  changes  with 
the  nature  of  the  demand* 


*  See  Boxer,  P.J.  (2008)  What  Price  Agility?  Managing  Through-Life  Purchaser-Provider  Relationships  on  the  Basis  of  the  Ability  to  Price  Agility , 
Navigator  White  Paper,  Software  Engineering  Institute,  Carnegie  Mellon  University,  September  2008 
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Making  the  two  models  coexist 


Talking  about  irregular  or  asymmetric 
warfare  and  institutionalizing 
counterinsurgency  skills,  ... 

•  How  do  we  institutionalize  procurement 
of  such  capabilities  -  and  the  ability  to 
get  them  fielded  quickly? 

•  Why  do  we  have  to  go  outside  the  normal 
bureaucratic  process? 


The  challenge  is  whether  in  our 
bureaucracy  and  in  our  minds  these  two 
different  paradigms  can  be  made  to 
coexist. 

•  The  key  is  to  make  sure  that  the  strategy 
and  risk  assessment  drives  the 
procurement,  rather  than  the  other  way 
around. 


•  These  forms  of  warfare,  skills 
and  abilities  demand  Type  III 
Agility. 

•  This  means  modernization 
‘+\  in  which 

-  Campaign  Strategy  and 
Interoperability  Risk 
Assessment  drive 
procurement. 

-The  full  Double  ‘V’  cycle  is 
managed  to  create  value  for 
Defense. 

-  Suppliers  support  different 
value  equation  models  on  the 
value  stairs  depending  on  the 
nature  of  the  demand. 


Extracted  from  speech  delivered  by  Secretary  of  Defense  Robert  M.  Gates, 
National  Defense  University,  Washington,  D.C.  September  29,  2008 


Software  Engineering  Institute 


Carnegie  Mellon 


The  Value  Stairs: 
changing  the  value  equation 
Philip  Boxer,  October  21st  2008 

©2008  Carnegie  Mellon  University 


23 


Contact  Information 


Philip  Boxer 


Research,  Technology  and  Systems  Solutions  Program, 
Software  Engineering  Institute,  Carnegie  Mellon  University 


Email:  pboxer@sei.cmu.edu 
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Software  Engineering  Institute 
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Abstract 


1.  New  kinds  of  threat  and  much  wider  varieties  of  demand  on  mission  capabilities  are  requiring  the 
military  to  achieve  unprecedented  levels  of  agility  and  responsiveness,  and  are  driving  the 
transformation  of  military  capabilities. 

2.  The  great  benefit  of  net-enablement  in  this  new  strategic  environment  is  that  it  enables  mission 
capabilities  to  be  orchestrated  and  composed  from  constituent  capabilities  within  the  context  of  systems 
of  systems. 

3.  The  presentation  will  outline  three  essential  ways  in  which  the  foundational  nature  of  the  systems 
engineering  task  needs  to  be  transformed  to  take  advantage  of  these  new  possibilities,  and  will  use 
examples  from  various  military  contexts  to  illustrate  their  applicability. 

•  First,  the  definition  of  systems-of-interest  also  has  to  give  an  explicit  account  of  the  contexts-of-use  from  which 
emerge  new  forms  of  demand  for  mission  capability. 

•  Second,  the  definition  of  systems-of-interest  has  to  be  extended  to  include  their  socio-technical  nature. 

•  Third,  it  has  to  be  possible  to  analyze  how  these  new  forms  of  demand  translate  into  new  patterns  of  interoperability 
(geometries-of-use)  across  systems  of  systems,  thus  defining  the  agility  of  systems  of  systems  in  terms  of  the 
required  varieties  of  geometry-of-use  that  they  must  support. 

4.  The  presentation  will  conclude  by  considering  the  impact  this  has  on  the  suppliers’  role,  the 
acquisition  process,  and  in  particular  the  changes  it  introduces  into  how  value  is  defined. 
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NO  WARRANTY 


THIS  CARNEGIE  MELLON  UNIVERSITY  AND  SOFTWARE  ENGINEERING  INSTITUTE 
MATERIAL  IS  FURNISHED  ON  AN  “AS-IS"  BASIS.  CARNEGIE  MELLON  UNIVERSITY 
MAKES  NO  WARRANTIES  OF  ANY  KIND,  EITHER  EXPRESSED  OR  IMPLIED,  AS  TO 
ANY  MATTER  INCLUDING,  BUT  NOT  LIMITED  TO,  WARRANTY  OF  FITNESS  FOR 
PURPOSE  OR  MERCHANTABILITY,  EXCLUSIVITY,  OR  RESULTS  OBTAINED  FROM 
USE  OF  THE  MATERIAL.  CARNEGIE  MELLON  UNIVERSITY  DOES  NOT  MAKE  ANY 
WARRANTY  OF  ANY  KIND  WITH  RESPECT  TO  FREEDOM  FROM  PATENT, 
TRADEMARK,  OR  COPYRIGHT  INFRINGEMENT. 

Use  of  any  trademarks  in  this  presentation  is  not  intended  in  any  way  to  infringe  on  the  rights 
of  the  trademark  holder. 

This  Presentation  may  be  reproduced  in  its  entirety,  without  modification,  and  freely 
distributed  in  written  or  electronic  form  without  requesting  formal  permission.  Permission  is 
required  for  any  other  use.  Requests  for  permission  should  be  directed  to  the  Software 
Engineering  Institute  at  permission@sei.cmu.edu. 

This  work  was  created  in  the  performance  of  Federal  Government  Contract  Number  FA8721- 
05-C-0003  with  Carnegie  Mellon  University  for  the  operation  of  the  Software  Engineering 
Institute,  a  federally  funded  research  and  development  center.  The  Government  of  the  United 
States  has  a  royalty-free  government-purpose  license  to  use,  duplicate,  or  disclose  the  work, 
in  whole  or  in  part  and  in  any  manner,  and  to  have  or  permit  others  to  do  so,  for  government 
purposes  pursuant  to  the  copyright  license  under  the  clause  at  252.227-701 3. 
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Overview 


Mission  Area  Analysis  Today  -  JCIDS 
Mission  Area  Analysis  circa  1960’s 

•  The  A-X  Example 

A-X  Concept  Formulation 

Comparison  and  Contrast 

Air  Force  Center  for  Systems  Engineering 
Case  Studies 


(U)  NORTHROP 

Ft COMMANDED  DESIGN  GENERAL  ARRANGEMENT 
(Figure?  UNCLASSIFIED) 


Decisions  and 
Decision  Making* 


Decision  -  A  Definition: 

1 .  A  choice  from  among  a  set  of  alternatives 

2.  An  irrevocable  allocation  of  resources 

Steps  in  the  Decision  Making  Process: 

1 .  Formulation  of  preferences  that,  for  the  situation  at  hand,  define 
good  and  bad  and  differentiate  levels  of  goodness 

2.  Generation  of  a  set  of  alternatives  for  consideration  of  choice 

3.  Evaluation  of  alternatives  against  the  decision  maker’s  preference 

4.  Selection  of  the  preferred  alternative  in  accordance  with  the 
decision  maker’s  preference 


*  Drawn  from  several  papers  by  G.  Hazelrigg,  appearing  in 
the  ASME  Journal  of  Mechanical  Design 


Decision  Making  in 
Conceptual  Design 


What  are  the  operational  capabilities  that  are  needed? 

Should  a  conceptual  design  effort  be  undertaken? 

What  mix  of  systems  (legacy  and  new)  are  likely  to 
achieve  the  desired  operational  capabilities? 

For  materiel  approaches  (new  systems),  which  system 
concept  (usually  a  mixture  of  technologies)  should  be 
the  basis  of  the  design? 

Which  technology  for  a  given  subsystem  should  be 
chosen? 

What  existing  hardware  and  software  can  be  used? 

Is  the  envisioned  concept  technically  feasible,  based 
on  cost,  schedule  and  performance  requirements? 

Should  additional  research  be  conducted  before  a 
decision  is  made? 


W  JCIDS*  Analysis 

^r‘ 


What  is  an  Architecture? 


“The  structure  of  components, 
their  relationships,  and  the 
principles  and  guidelines 
governing  their  design  and 
evolution  over  time.” 

(IEEE  STD  610.12  as  stated  in 
the  DoD  Architecture 
Framework  (DoDAF) 


APXITEKTQN  (Greek)  =  Master  Builder 


Is  JCIDS  sound  policy? 


Recall  our  decision  making  process.... 

1 .  Formulation  of  preferences  that,  for  the  situation  at  hand, 
define  good  and  bad  and  differentiate  levels  of  goodness 

FAA-  Establish  Tasks,  Conditions,  Attributes  and  Measures 

2.  Generation  of  a  set  of  alternatives  for  consideration  of 
choice 

FNA  considers  current  alternatives 
Early  FSA  identifies  future  alternatives 

3.  Evaluation  of  alternatives  against  the  decision  maker’s 
preference 

FSA  -  Evaluates  alternative  approaches  against  FAA  criteria 

4.  Selection  of  the  preferred  alternative  in  accordance  with 
the  decision  maker’s  preference 

Concept  Decision  based  on  FSA  priorities  and  recommendations 


This  actually  makes  sense  when  you  consider  what  is  supposed  to  be  done! 


W  Is  JCIDS  Really  New? 


•  The  initial  instruction  (and  manual)  came  out  in  2003, 
but  is  it  really  new? 

•  Let’s  take  a  trip  back  in  time  -  approximately  40  years 
-  to  the  Close  Air  Support  challenges  of  the  1 960’s 


W  Lessons  from  Vietnam 


Air  Force  largely  unprepared  for  Close  Air  Support  (CAS) 
mission 

•  A-1 ,  A-37  had  insufficient  payload,  loiter 

•  Incompatible  comm  with  ground  units 

Army  doctrine  evolving  towards  air  mobile  tactics 

•  Increased  reliance  on  armed  helicopters 

•  Initiated  development  of  AH-56  Cheyenne 

Johnson-McConnell  Agreement 


AF  retained  CAS  mission,  but  recognized  role  of  Army 
helicopters  for  fire  support 

Army  gave  up  large  fixed-wing  transports 


Task  Definition 


Three  Mission  Tasks 

•  Close  Support  Fire  (CSF) 

•  Armed  Escort  (AE) 

•  Armed  Reconnaissance  (AR) 

•  CSF  and  AE  were  considered  complementary 

•  AR  involved  different  weapons  and  acquisition 
systems,  considered  a  secondary  A-X  mission  due 
to  parallel  development  of  AC-130  gunship 


The  System  of  Systems 
Perspective 


A/AF  SYSTEM  PREPLANNED  REQUESTS 


Copojnand  and  Control 


tacc 


Ground  Alert 


Scheduled  | 


Scramble 


Takeoff 


r 


DASC 


Enroute 


CRC 

~r~ 


Debriefing 

ZTZ 


Landing 


CRP 


Air  A 


lert  ) 


RDZ 


Fwd.  Base  \ 

V- 


*CSF 


1 


Re turn 


FAC 


Target 

Designation 

- E - 


*AE 


*AR 


CAS  Task  »^Tgt.  Acq\|— ►  Wpn  -  Del. 


Post-Strike 

Assessment 


K 


proceed  to 
Another 
Ta  rget 


Target  Acquisition 

l  ~ 


Battalion  Request 
for  Air  support 


Coordinates  with 
S-2 ,  S-3,  ARTY  Lo,  FAC 


TACP 


Brigade  Processes 
Request  for  Air  Support 


Coordinates  with 
S-2,  S-3,  ARTY  LO,  FAC 


Division  Processes 
Request  for  Air  Support 


Coordinates  with 
G-2  *  G-3,  FSC ,  ALO 


_ i 


Corps  Processes 
Request  for  Air  Support 


Coordinates  with 
G-2 ,  G-3,  FSC,  ALO 


TACC  Consolidates  Other 
Air  Requests  and  Frags 
Missions 


Coordination  for  Pre-Planned  CAS  Requests 


Return  to 
Air  Alert 


*AR^armed  reconnaissance;  AE — armed  escort;  CSF* — close  support  fires 
**This  sequence  occurs  every  time  with  CSF  and  to  varying  degrees 
with  AR  and  AE. 

The  Tactical  Air  Control  System  (circa  1968) 


But  aren’t  these  simply 
elements  of  a  mission 
architecture? 


^  ElK  5 ft  m 


^  p 

Attributes  and  Measures 


Only  four  key  mission  characteristics  specified  ! 

•  Responsiveness  considered  not  just  speed,  but  basing 
locations,  availability,  loiter  time  over  target,  and  ability  to 
communicate  with  ground  elements 

•  Simplicity  emphasized  ease  of  production,  maintenance, 
and  low  cost 

•  Lethality  made  it  clear  that  it  was  not  an  aircraft 
development  effort,  it  was  a  weapon  system  development 

•  Survivability  concerns  would  drive  redundancy, 
component  placement,  protection  systems, 
maneuverability,  targeting  systems,  et.al. 

•  Mission  characteristics  drove  performance  parameters, 
which  resulted  in  concept  aircraft  configurations 

•  Alternatives  evaluated  against  mission  and  cost  effectiveness 
measures 


Impact  of  Loiter  Time  and  Sortie  Rate  on  Force  Requirements 


A PM  26-3 
PLANNING  MMH/FH 

ACTUAL  SEA 
MMH/FH  AVERAGE 

P-4 

30 

33*2 

*F-105 

40 

27  *  6 

F-100 

25 

26*6 

F-5 

17 

15.5 

A— 1 

10 

14*3 

A-37 

7 

7*8 

Roll-in,  Delivery,  Pull-up  Maneuver 


Relative  Aircraft  Attrition  Versus  Velocity  and  Maneuver 


Maintenance  Man  Hours/Flight  Hour 
for  Vietnam  era  Aircraft 


^Capability  of  Existing  Systems 


•  F-4,  F-1 1 1  were  the  Air  Force’s  primary  tactical  aircraft 
of  the  time 

•  Both  were  expensive,  and  ill  suited  to  CAS  mission 

•  F-5 

•  Initially  the  Air  Force  choice  for  a  low-cost  tactical  fighter 

•  Better  air-to-air  capability  than  A-7 

•  A-7D 

•  Derivative  of  existing  Navy  aircraft 

•  Favored  by  many  in  OSD,  Congress 

•  Could  not  carry  a  big  gun,  significantly  lower  loiter  time 

•  Would  eventually  be  involved  in  a  flyoff  with  A-1 0  prior  to 
production  decision 

•  Army  Helicopters? 

•  Roles  and  missions  agreements  prevented  serious  consideration 


W  Aircraft  Comparison 

Mr 


■ 


A-lJ 

OV-10 

(Impr.) 

A-37B 

A-X 

A-7D 

F-4C 

Operating  weight  empty  (lb) 

(includea  crew,  gun,  ammunition) 

13,328 

9,44b 

6,200 

20,140 

19,250 

31,097 
w/gun  pod* 

Internal  fuel  capacity  (lb) 

2,280 

3,680 

2,974 

7,000 

9,750 

12,818 

External  load  capacity — with  FIF 
(lb) 

9,392 

4,  394 

4,826 

16,860 

14,000 

14,085 

«/' Maximum  TOGW  (lb) 

25,000 

17,514 

14,000 

44,000 

43,000 

58,000 

'  Engines  (number /type) 

one 

R-3350 

two 

T-76 

two 

J-85 

two 

T-55 

one 

TF-41 

two 

J-79 

'lawful  load  capacity  (fuel  and 
ordnance^lba)  for  takeoff — 
distance  (Ground  Run,  S.L., 

Tropic  .Day)  of : 

750  ft 

1,000  ft 

4 , 000** 

6 , 2doV* 

1,30Q** 

3,600 

3, 200** 
4,000** 

9,000 

12,500 

-0- 

-0- 

-0- 

-0- 

Maximum  speed,  clean,  S.L.  (KTAS) 

277 

262 

417 

400 

607 

M  1.2 

Beat  cruise  speed,  5,000  ft, 
maximum  ordnance  (KTAS) 

170 

170 

265 

240 

315 

420 

Ferry  range,  unrefueled  (NM) 

2,800 

2,600 

1,560 

2,600 

2,600 

1,600 

Number  of  ordnance  stations 

15 

7 

8 

10 

8 

5 

Internal  guns  (number /caliber) 

four 

20 -mm 

four 

7 .62-mm 

one 

7 .62-mm 

one 

30 -mm 

one 

20 -mm 

*  (one 
SUU-16 

20 -mm 
pod) 

♦•Cannot  land  in  thia  distance  at  any  weight. 


A-X  Concepts 


Requirements  from  Dec  1966 
Requirements  Action  Directive 


Performance  Parameter 

Desired 

Required 

Gross  Weight  (lbs) 

22,500 

30,000 

Payload  -  Mixed  Ordnance  (lbs) 

8,000 

6,000 

Combat  Radius  (nautical  miles) 

--- 

200 

Loiter  Time  @  Combat  Radius  (hrs) 

--- 

2 

Min  Maneuvering  Speed  @  5000  ft  (knots) 

120 

150 

Turn  Radius  @  Combat  Weight  (ft) 

1,000 

2,000 

Max  Speed  @  Sea  Level  w/  Ext.  Ordnance  (knots) 

550 

450 

•  Concept  design  studies  conducted  in  1967 

•  Resulted  in  two  government  configurations,  and  four  contractor 
configurations 

•  Concept  determined  to  be  feasible  within  existing 
technology 

•  Most  configurations  used  turbo-prop  designs 

•  Identified  risk  elements  included  gun/ammunition  development 
and  integration,  and  early  IOC 

•  Lean  avionics  packages  defined  to  keep  costs  down 

•  Concept  Formulation  Package  (predecessor  to  Initial 
Capability  Document)  completed  in  1968 


A-X  Concepts 


■ 


(U)  GRUMMAN 

RECOMMENDED  DESIGH  GENERAL  ARRANGEMENT 
(Figure  UNCLASSIFIED) 


(U)  GENERAL  DYNAMICS 
RECOMMENDED  DES IGN  GENERAL  ARRANGEMENT 


(Figure  UNCLASSIFIED) 


(u)  McDonnell  dougias 
recommended  design  general  arrangement 


(Figure  UNCLASSIFIED) 


(now  referred  to  as  Concept  Refinement) 

•  Single  or  twin  turboprop  propulsion  gave  way  to  twin  turbofan 

(leveraged  Navy  S-37  aircraft  development) 

•  Payload  essentially  doubled  to  16,000  lbs  -  led  to  aircraft  size/cost  growth 


V  JCIDS  40  Years  Early? 


Did  the  A-X  concept  formulation  adhere  to  (in  retrospect) 
JCIDS  principles? 

Yes  ,  kind  of ... 

•  Clear  definition  of  tasks,  conditions  and  measures  (FAA) 

•  Consideration  of  a  range  of  existing  Air  Force  systems  to 
provide  the  needed  capability  (FNA) 

•  Concept  formulation  traceable  to  previously  defined  tasks, 
conditions  and  measures  (FSA) 

Shortcomings 

•  No  serious  consideration  of  the  full  range  of  joint  warfighting 
concepts  to  meet  the  capability  needs 


Summary 


•  The  A-X  concept  formulation  was  rigorous  and  traceable 
to  user  needs 

•  While  full  consideration  of  joint  concepts  may  not  have 
been  done,  the  emphasis  was  not  on  joint  capabilities 

•  Aircraft  has  performed  well,  and  is  still  in  service  today 


C-5  Galaxy 


A-10 


F-111  Aardvark 


Peacekeeper  Intercontinental 
Ballistic  Missile 


Hubble  Space  Telescope 


GPS  (Global 
Positioning  System) 


B-2 


TBMCS  (Theater  Battle 
Management  Core  Systems) 


Website: 

http://www.afit.edu/cse/ 


W  Ongoing  &  Future  Case  Studies 


International  Space  Station 


Underway 
Global  Hawk 


Underway 


MH-53J/M  Helicopter 


FY09  Option 
KC-135  Simulators 


FY09  Start 


E-10 


FY10  Option 
T-6A  Texan  II 


FY10  Start 


Questions? 
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Overview 


•  Systems  Engineering  in  Sustainment  Phase 

•  A-10  Development  and  Operational  Service 

•  Aircraft  Structural  Integrity  Program 

•  Structural  Problems  on  the  A-10 

•  HOG-U P/Service  Life  Extension 

•  Re-winging  Decision  and  the  A-10C 

•  Summary 


\J  SE  Sustainment  Activities 


A  Partial  List: 

•  Execution  of  strategies  for  operations,  sustainment  and, 
when  necessary,  disposal 

•  Maintain  baselines,  data,  and  supply  chain 

•  Maintain  Operational  Suitability,  Safety  and 
Effectiveness 

•  Monitoring  and  comparing  performance  and  condition  to  design 
and  prediction  models 

•  Re-engineering  of  legacy  system  performance 
requirements  and  designs 

•  Decision  analysis  support  for  upgrades/mods  and  life 
extension  decisions 

•  May  include  modifications  to  maintenance  concepts 


Aircraft  Structural 
Integrity  Program  (ASIP) 


•  ASIP  Initiated  in  1958 

•  Monitor  and  evaluate  structural  health  of  AF  aircraft 

•  AFI-63-1001  requires  plan,  MIL-HDBK  1530  provides 
guidelines  and  details 

•  During  1970’s  and  80’s 

•  Damage  Tolerance  Assessments  (DTA) 

•  Inspection  and  modification  programs 

•  Fatigue  tests  on  wing,  fuselage,  and  full  aircraft 

•  Used  to  develop  individual  aircraft  tracking  program,  and 
tech  orders  for  inspection,  maintenance  and  repair  actions 


A-10:  Early  Struggles 


•  Within  the  Air  Force 

•  Close  Air  Support  (CAS)  was  considered  less  important 
than  strategic  bombing,  air  superiority,  and  interdiction 

•  Tactical  force  mix  required  less  expensive  aircraft,  but  AF 
still  favored  fast  multi-role  fighters 

•  F-5,  A-7D  were  early  choices  for  the  CAS  role 

•  Reluctantly  agreed  to  pursue  specialized  CAS  aircraft 


A-10:  Early  Struggles  (ctd.) 


Within  the  Army 

•  Unsatisfied  with  level  of  CAS  provided  by  Air  Force 

•  Doctrine  evolving  towards  air  mobile  tactics 

•  Increased  reliance  on  armed  helicopters 

•  Initiated  development  of  AH-56  Cheyenne 

•  Competed  with  AF  for  CAS  development  $ 


Johnson-McConnell  Agreement  (1966) 

•  AF  retained  CAS  mission,  but  recognized  role  of  Army 
helicopters  for  fire  support 

•  Army  gave  up  large  fixed-wing  transports 


A-10:  The  aircraft  that 
almost  wasn’t! 


•  Survivability  -  redundancy,  shielded  systems,  engine  placement 

•  Maintainability  -  interchangeable  left/right  side  parts,  simple  skin 
panels,  engine  placement 

•  Cost  Considerations  -  lean  avionics  (no  night/adverse  weather 
systems),  ammunition  cost  reduction  efforts 


A-10  Deployment,  and  Debate  ! 


•  Final  production  aircraft  delivered  in  1984 

•  No  service  support  for  continued  production  (F-16  factor) 

•  Army  Air-Land  Battle  doctrine 

•  Greater  reliance  on  Battlefield  Air  Interdiction  (BAI) 

•  Survivability  concerns  associated  with  greater  SAM  threat 

•  By  1985,  studies  emerged  suggesting  an  A-16  as  a 
replacement  for  the  A-10 

•  Defense  Authorization  Act  for  FY88-89 

•  Directed  completion  of  CAS/BAI  Master  Plan 

•  Directed  yet  another  CAS  fly-off 
(A-10,  F-16,  A-7,  AV-8,  F/A-18) 


Desert  Storm 


•  High  effectiveness,  and  demonstrated  survivability 

•  High  sortie  rate,  low  maintenance  man  hours/flight  hour 

•  CAS  F-16’s  performed  poorly,  reverted  back  to  standard 


•  Post  war  decisions 

•  Serious  proposal  floated  by  CSAF  to  give  CAS  and  A-10  to 
Army  in  exchange  for  ATACMS,  space  mission,  et.al. 

•  AF  decided  to  keep  A-10,  but  in  reduced  numbers 


A-10  Structural  Configurations 


Retrofit  WOP 
Configuration 

Intended  for  Aircraft 
7-441  (not  completed 
on  all  aircraft) 

Thin  wing  center  panel,  cold  worked  at 
WS  0,  Retrofit  thick  wing  outer  panel. 
Qualified  to  6,000  hours  Spectrum  3. 

Production 

WOP 

Aircraft  442-581 

Thin  wing  center  panel,  cold  worked  at 
WS  0,  Production  thick  wing  outer  panel. 
Qualified  to  6,000  hours  Spectrum  3. 

Thick  Skin 
Configuration 

Aircraft  582  and 
subsequent 

Production  increased  wing  center  panel 
and  outer  panel  thickness. 

Configuration  qualified  to  8,000  hour 
service  life. 

Notes: 

•  Original  design  life  was  6,000  flight  hours 

•  Design  load  spectrum  changed  in  1977  based  on  measured  fleet  usage 

•  Fatigue  test  failed  at  less  than  60%  of  new  spectrum  service  life 

•  Resulting  production  and  retrofit  changes  indicated  above 


ASIP  Implementation 


•  Fairchild  sold  A-10  rights  to  Grumman  in  1987 

•  Fairchild  ceases  to  exist  shortly  after 

•  Grumman  delivers  updated  DTA  and  associated 
Force  Structural  Maintenance  Plan  (FSMP) 

•  Never  fully  incorporated  into  tech  orders,  not  accomplished 

•  Difficulty  with  field  inspections,  budget  constraints  cited 

•  Analytical  Condition  Inspection  (ACI) 

•  Addressed  some  inspection  locations,  but  on  few  aircraft 

•  Cracks  found  in  several  locations  in  1995,  96 

•  Cracks  classified  as  minor 


•  1994  -  Northrop  merges  with  Grumman 

•  Although  NG  still  the  prime,  most  mods  competed  or  done  organically 
by  government 

•  “Fallout  funds  used  to  task  NG  to  incorporate  design  changes  into 
configuration  baseline  drawings...” 

•  1995  Base  Realignment  and  Closure  (BRAC) 

•  Closes  McClellan  AFB 

•  Maintenance  and  repair  operations  moved  to  Hill  AFB 

•  Results  in  loss  of  80%  of  experienced  workforce  by  2000 

•  1 997  -  SPO  competes  prime  sustainment  contract 

•  Lockheed  Martin  Systems  Integration  wins 

•  NG  expected  to  be  part  of  team  due  to  proposed  LM-NG  merger 

•  1 998  -  LM-NG  merger  called  off 

•  NG  reduced  to  supporting  role 


HOG  UP 


•  1998:  Northrop  Grumman  delivers  “A-10A  Aircraft  Wing 
Center  Panel  Rework-Fatigue  Life  Improvement”  report 

•  Detailed  changes  required  to  support  16,000  hour  service  life 

•  Based  on  assumption  that  1993  FSMP  implemented 

•  1999:  SPO  initiates  HOG  UP 

•  Repair  program  vice  modification 

•  Allowed  use  of  maintenance  funding 

•  Did  not  require  acquisition  approval 

•  Configuration  Control  Board  action  not  required 

•  HOG  UP  expands  to  catch  other  necessary  changes 

•  No  composite  assessment  of  structural  risk 

•  Cost  growth  from  $140M  to  $600M,  not  including  unprogrammed 
cost  for  WS-23  inspection  and  repair 

•  No  full-scale  fatigue  test  to  validate  HOG  UP 


HOG  UP  Evolution 


w 


Hog  Up  1999  and  2003 


4  j  Sometimes,  things  have  to  get 
^  worse  before  they  can  get  better! 


•  HOG  UP  delays  due  to  WS-23  inspection  and  repair 

•  Number  of  unusable  wings  higher  than  expected 

•  Predictions  that  serviceable  wings  would  run  out  by  201 1 

•  Back-up  of  aircraft  in  depot  due  to  longer  than  expected  repair  times 

•  Catastrophic  failure  of  HOG  UP  wing  in  fatigue  test  (2003) 

•  Well  short  of  16,000  hour  life  expectancy 

•  2005:  AF  completes  business  case  analysis 

•  Option  1 :  Organic  sustainment  of  thin  skinned  wings,  increase  SLEP 
for  all  wings  ($4.6B) 

•  Option  2:  Buy  135  wings,  increase  SLEP  for  remaining  wings  ($3.16B) 

•  Option  3:  Buy  242  wings  and  avoid  cost  of  SLEP  ($1 .72B) 

•  2006:  AF  competes  contract  for  new  wings!  (Option  3) 

•  Boeing  wins  contract  to  build  wings,  to  be  installed  on  a  Fairchild 
Republic  aircraft,  being  maintained  by  Lockheed  Martin! 


Learning  Principle  5* 


Successful  design,  development  and  production  is  not 
enough  to  sustain  a  system  throughout  its  life  cycle. 

•  A-10  sustainment  efforts  were  severely  impacted  by  a 
number  of  factors 

•  On-again,  off-again  retirement  decisions 

•  Vanishing  prime  contractor 

•  BRAC,  and  general  turnover  of  government  personnel 

•  Loss  of  condition  baseline  led  to  initially  poor  decisions 
regarding  life  extension  efforts 

•  A-10  sustainment  has  recovered,  but  after  significant  cost 
associated  with  the  original  HOG-UP  program 


*  6  Learning  Principles  are  contained  in  the  A-10  Case  Study 


A  Second  Life  for  a 
Modern  Day  Hog 


•  Low  Altitude  Safety  and  Targeting  Enhancements  (1990’s) 

•  Embedded  GPS/INS  system  added  (1999) 

•  Precision  Engagement  (2005) 

•  Results  in  A-10C  Designation 

•  Replacement  of  TF-34  Engines  (Proposed) 


C-5  Galaxy 


A-10 


F-111  Aardvark 


Peacekeeper  Intercontinental 
Ballistic  Missile 


Hubble  Space  Telescope 


GPS  (Global 
Positioning  System) 


B-2 


TBMCS  (Theater  Battle 
Management  Core  Systems) 


Website: 

http://www.afit.edu/cse/ 


W  Ongoing  &  Future  Case  Studies 


International  Space  Station 


Underway 
Global  Hawk 


Underway 


MH-53J/M  Helicopter 


FY09  Option 
KC-135  Simulators 


FY09  Start 
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FY10  Option 
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Notional  View  of 
Software  Measurement 


Generate  SW 

Earned  Value 

appropriate 

Management  (EVM) 

Determine  methods  of 
obtaining  cost  estimating 

WBS 

for  SW 

data 

Use  estimation  tools, 
techniques, 
processes,  &  practices 

Link  SW  quality 
indicators  to  EVM 

Concepts  -  Requirements  -  Arch/Design  Development  -  Maintenance 


Software  Engineering  and  Systems  Assurance  (SSA)  initiatives 

•  Software  Resources  and  Data  Report:  Feasibility  Study 

•  Revision  of  MIL-HDBK-881 A  to  improve  software  guidance 

•  Program  feasibility  analysis  using  estimation  models 

•  Integration  of  software  metrics  with  EVM  to  assess  consistency 
of  estimates 
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Ml  L-HDBK-881A 


•  Military  Handbook  881A  is  the  Department  of 
Defense  handbook  on  Work  Breakdown 
Structures  (WBS)  for  Defense  Materiel  Items 

-  A  WBS  provides  a  consistent  and  visible  framework  for  defining 
work  and  structuring  contracts  within  a  program 

-  Approved  guidance  for  DoD  Departments  &  Agencies 

-  Current  version  was  released  on  30  July  2005 

-  MIL-HDBK-881 A  is  controlled  by  the  Office  of  the  Undersecretary 
of  Defense  (Acquisition,  Technology,  and  Logistics)  (OUSD 
(AT&L))  Acquisition  Resources  and  Analysis  (ARA) 


MIL-HDBK-881A  due  for  update  consideration 
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Software  in  Mi L-HDBK-881A 


•  SSA  initiated  a  Software  Cost  Control  Working  Group 
project  to  provide  software  recommendations 

-  Including  representation  from  the  Services,  DCMA,  ARA,  DAU, 
NDU,  PA&E/DCARC  and  using  NDIA  software  experts  panel 

•  MIL-HDBK-881A  revision  objectives: 

-  Make  handbook  acceptable  of  software  engineering  practice 

-  Correct  errors  and  inconsistencies 

•  Walkthrough  of  MIL-HDBK-881A  revealed 
inconsistencies  with  respect  to  defense  material  items 
and  software  intensive  systems  development 

-  Handling  of  System  Development  &  Demonstration  (SDD) 
phase  software  engineering  activities  is  insufficient 

-  Decision  made  to  provide  revisions  versus  complete  rewrite 
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Comment  Summary 


•  Notable  changes: 

-  Replace  'material  item'  with  ' 
acquisition  program' 

-  Add  words  to  make  'artifacts' 
equal  to  'products' 

-  Include  words  that  make  'product- 
oriented'  and  'DoDAF  architecture' 
views  acceptable  WBS  hierarchy 
structures 

•  Results 

-  Compiled  into  Comment  matrix 

-  Drafted  a  new  Appendix  B  for 
software 


Comments  by  Severity 


Critical  Subst  Admin 


Critical  comments  directly  support 
revision  objectives 
Substantive  comments  highlight 
incorrect,  misleading, 
potentially  unnecessary,  or 
inconsistent  text 

Administrative  comments  captures 
typos,  paragraph  structure,  etc. 
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Example  #1  Revision 


Comment  Matrix  Entry  #16:  Paragraph  1.7  WBS  Evolution: 

“For  material  item  acquisitions,  Smee-the  system  is  mainly  a  concept 
at  this  point,  it  is  not  until  the  System  Development  and 
Demonstration  (SDD)  phase  that  the  system  is  broken  into  its 
component  parts  and  a  detailed  WBS  can  be  developed.  In  the 
SDD  phase,  configuration  items  that  describe  the  Program  WBS  are 
first  identified  and  contracts  can  be  awarded  to  develop  these  items. 
By  the  end  of  SDD,  the  WBS  is  fully  defined  to  its  lowest  levels  that 
best  represents  the  system. 

For  software  intensive  systems  and  acquisition  programs  that 

involve  procuring  in  single  or  very  low  volume,  there  needs  to  be  a 

greater  refinement  of  the  engineering  activities  in  the  Technology 

Development  phase  within  the  Program  WBS.  For  these  types  of 

acguisition  programs,  it  is  essential  that  both  government  and 

contractor  can  agree  on  a  fully  defined  WBS  at  Milestone  B,  prior  to 

entering  SDD.  “ 
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Example  # 2  Revision 


Comment  Matrix  Entry  #25:  Delete  Paragraph  2.3.1 
Specifications  and  Drawings: 

“  The  family  of  specifications  and  drawings,  resulting  from 
the  progressive  steps  of  the  systems  engineering  process, 
provides  the  basis  for  the  Program  WBS,  the  Contract 
WBS,  and  its  extensions.” 

Rationale:  For  software  intensive  systems,  specifications  and 
drawing  are  products  normally  produced  after  PDR  which  is  too 
late  to  drive  the  development  of  the  initial  Program  WBS. 
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Overview  of  New  Appendix  B 


•  Leveraged  text  from  draft  MIL-HDBK-171 

•  Contains  three  WBS  examples  to  encourage  thoughtful 
tailoring  based  on  project  characteristics 

•  Emphasized  use  of  standards  and  consistent  use  of 
terminology  when  defining  WBS  elements 

•  Maintained  Appendix  ‘look  and  feel’  as  the  other 
Appendices 

•  Included  ‘notes’  to  provide  guidance  on  handling  COTS 
and  software  development  methodologies 
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Ml L-HDBK-881A  Project  Summary 


•  Working  group  met  our  goal  to  provide  a  software 
community-coordinated  set  of  recommendations  to 
OUSD  AT&L  ARA  as  they  began  official  review  and 
update  process 

-  Maintained  ‘software’  focus 

•  Reached  out  to  industry  members  of  NDIA  to  review  the 
suggested  changes 

-  Validated  recommended  changes  are  improvements  from 
industry  perspective 

-  Obtained  additional  examples  to  include  in  Appendix  B 


•  Next  steps  will  be  determined  based  on  results  of  ARA 
update  (i.e.,  contents  of  MIL-HDBK-881B) 
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The  System  Architecture 
Tradeoff  Analysis  Method® 
(SySATAM®) 


Mike  Gagliardi  and  Bill  Wood 


®  Architecture  Tradeoff  Analysis  Method  and  ATAM  are  registered  in  the  U.S. 
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Purpose  of  the  System  ATAM  -  1 


The  System  ATAM  is  a  method  that  helps  stakeholders  ask  the 
right  questions  to  discover  potentially  problematic  architectural 
decisions  (risks) 

Discovered  risks  can  then  be  made  the  focus  of  mitigation 
activities — for  examples: 

•  changing  architecture 

•  further  analysis 

•  extending  prototyping. 

Tradeoffs  can  be  explicitly  identified  and  documented 

•  Tradeoffs  made  already 

•  Upcoming  tradeoffs 


=  Software  Engineering  Institute  Carnegie  Mellon 
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Purpose  of  the  System  ATAM  -  2 


The  purpose  is  NOT  to  provide  precise  analyses. .  .  the  purpose 
IS  to  discover  risks  created  by  architectural  decisions. 

We  want  to  find  trends:  correlations  between  architectural 
decisions  and  predictions  of  system  properties. 


1  Software  Engineering  Institute  Carnegie  Mellon 
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Presentation  Outline 


What  is  an  ATAM? 


Similarities  and  Differences  between  ATAM  and  System  ATAM 


Highlights  of  Differences 


Experiences  and  results 


—  Software  Engineering  Institute  Carnegie  Mellon 
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Phase  2  -  Stakeholders 


The  following  is  a  partial  list  of  potential  stakeholders: 


software  architect 

maintainer 

tester 

performance  expert 
security  expert 
project  manager 
customer  (buyers,  acquirers) 
application  builder 
system  administrator 
service  representative 
system  architect 


developer 

integrator 

standards  expert 

reliability/availability  expert 

safety  expert 

product  line  manager 

end  user 

mission  specialist/planner 
network  administrator 
domain  representative 
device  H/W  expert 


Software  Engineering  Institute 


Carnegie  Mellon 


©2007  Carnegie  Mellon  University 
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What  is  an  ATAM  -1 


Process 

•  Actors 

—  sponsor  (Program  management)  and  architects  (6) 

—  Lead  Evaluator  -  has  lead  evaluator  training 
—  Evaluation  team  (4)-  all  have  taken  ATAM  training  courses 
—  Stakeholders  (20) 

Schedule 


What  is  an  ATAM  -2 


Technical  Basis 

•  Business  and  Mission  Drivers 

•  New  threats,  capabilities,  technology,  automation,  legacy 

•  Scalability,  schedules,  budgets,  joint,  coalition,  FMS 

•  There  is  a  documented  software  architecture  (SAD,  UML 
Diagrams) 

•  Multiple  viewpoints,  views,  framework 

•  Quality  attributes  are  the  architecture  drivers 


Performance 

Availability 

Security 

Usability 

Sustainability 


avoid  too  slow,  too  late,  bottlenecks 

avoid  fragility  due  to  failures 

avoid  spoofing,  unauthorized  access 

avoid  operator  overload 

avoid  hard  to  update  functions  and  new  COTS 


Interoperability,  scalability,  extensibility  etc 
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What  is  an  ATAM  -3 


Technical  Basis  (Continued) 

•  Scenarios  represent  the  quality  attributes 

—  Stimulus,  environment,  response 

—  “  A  tank  commander’s  COP  shows  an  identified  threat,  he  has  authorization  to 
engage  the  threat,  when  it  comes  within  his  range  he  conducts  a  successful 
engagement  and  reports  it  via  the  COP”. 

—  Elicited  in  a  meeting  with  stakeholders  (or  from  previous  QAW) 

•  Architectural  approaches  can  be  identified  and  analyzed 

Passive  and  active  redundancy,  publish/subscribe,  client/server,  reliable  protocol 

•  Architectural  Decisions 

—  Provide  a  tool  to  assist  with  mapping  spectrum  allocation  to  force  structure 
—  Break  down  a  system  into  components  for  transportation 
—  Use  a  proxy-based  pub/sub 


- —  Software  Engineering  Institute  Carnegie  Mellon 
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What  is  an  ATAM  -  4 


Technical  Basis  (Continued) 

•  Walking  scenarios  through  the  software  architecture,  and  having  the  ATAM 
team  and  stakeholders  probe  the  quality  attributes  exposes  architectural 
risks  and  maps  each  risk  to  business  drivers 

•  These  risks  can  be  “rolled  up”  into  risk  themes  mapped  to  business  drivers 

Results-  content 

•  A  number  of  scenarios  (10  to  15)  are  analyzed  and  documented 

•  Table  of  risks,  trade-offs,  programmatic  issues,  atta-boys 

•  Rollup  of  the  risks  into  risk  themes 

Results-  documents 


•  Summary  Outbriefing  after  Stakeholder  Phase  (1  hour) 

•  Report  (50,  60  pages)  of  findings  with  an  Executive  Summary  (  2  pages) 
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Commonalties  and  Differences  -1 


The  System  ATAM  (including  software)  basically  conforms  to  the 
AT AM  process,  technology,  and  results  as  follows 


Process 

System 

Fast  Tracking  of  subject  matter  experts  (SME) 

SM  designers 

Phases 

More  careful  scoping  (what’s  in,  what’s  out) 

Need  system  (block  diagrams) 

views  and  white  papers 

Technical 

Quality 

Attributes 

A  few  additional  QA  (transportability,  shake  and  bake, 
force  modularity,  spectrum  management) 

Scenarios 

Stress  system  aspects  as  well  as  software 

Analysis 

Combination  of  system  and  software  architects 

System  Architectural  Approaches 

Results 

No  differences  in  either  the  outbriefing  or  the  report  | 

—  Software  Engineering  Institute  Carnegie  Mellon 
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Highlights  of  Experiences  -1 


ATAM 

•  Four  2  day  courses  providing  the  basic  software  architecture  knowledge, 
including  an  ATAM  team  lead  evaluator  course 

•  Have  conducted  numerous  ATAMS 

•  Have  an  ATAM  Reference  Guide  for  the  team 

•  Have  extensive  set  of  templates  to  assist  the  team  in  all  activities 

•  External  organizations  (commercial,  DoD  contractors)  have  qualified  leads 

SySATAM 

•  Have  a  process  in-place  for  conducting  SySATAMs 

•  Still  in  piloting  Phase-  have  conducted  2  SySATAMs 

•  Have  extensive  set  of  templates  to  assist  the  team  in  all  activities 


=  Software  Engineering  Institute  Carnegie  Mellon 
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Highlights  of  Experiences  -2 


SME  Experiences 

•  On  one  system  an  Evaluation  Team  member  was  also  an  SME 

•  On  the  other  the  SME  was  a  seasoned  Mechanical  Engineer  and  a  domain  expert 

—  Took  the  SME  training 

—  Evaluation  team  had  to  initially  prompt  the  SME  for  risks. 

New  Quality  Attributes  and  associated  risks 

•  Force  Modularity,  Mobility,  Spectrum  Management 

•  Logistics,  installation,  mechanical  checks 

New  Considerations 

•  DoDAF  operational  views 

•  experimental  simulation  and  analysis  results 

•  white  papers 

•  Manual  versus  automated  activities  are  more  prevalent 


—  Software  Engineering  Institute  Carnegie  Mellon 
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Highlights  of  Experiences  -3 


Architectural  Representations 

•  System  architecture  documentation  consists  mainly  of  block  diagrams  and 
sequence  diagrams  and  some  DoDAF  lower  level  views 

Stakeholders 

•  System  engineers  tend  to  trump  the  software  engineers 

•  Good  exercise  for  system  and  software  arch  and  eng  to  get  on  the  same 
page 

Surprises 

•  Preparation  phase  was  easier  than  expected,  scoping  was  straightforward 


—  Software  Engineering  Institute  Carnegie  Mellon 
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Highlights  of  Experiences  -  4 


Typical  Risk  Themes 

•  There  are  a  number  of  significant  system  engineering  issues  that 
require  further  analysis  as  a  basis  for  architectural  decision 

•  CONOPS  for  Using  Programs  has  not  been  updated/supplemented  to  take 
this  system  into  effect 

•  Architectural  support  for  flexibility  is  powerful.  However,  without  careful 
management  of  flexibility  it  could  become  overly  complex  and  impose  an 
unnecessary  cognitive  burden  on  users. 

•  Approach  to  automate  and  reduce  test  time  not  thought  out 

•  Fault  Tolerance  approach  needs  to  be  developed 


=  Software  Engineering  Institute  Carnegie  Mellon 


D2007  Carnegie  Mellon  University 
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Conceptual  Flow  of  ATAM 


impacts 


Business 

Drivers 

Quality 

Attributes 

Scenarios 

I  System  & 
Software 
Architecture 

Architectural 

Approaches 

Architectural 

Decisions 

T  radeoffs 


Sensitivity  Points 


Risk  Themes 


distilled 

into 


Non-Risks 


Risks 


-  Software  Engineering  Institute 


Carnegie  Mellon 


D2007  Carnegie  Mellon  University 


Conclusion 


System  ATAM  is  a  natural  extension  to  the  ATAM 

•  Basic  approach  works  just  fine 

SME  is  needed  with  functional/domain  expertize 

•  Fast  track  training  was  effective 

Risk  Themes  identified  areas  to  help  the  programs  choose  what  to 
explore  to  firm  up  the  architecture 

•  Both  software  and  system  risks  were  revealed 

Have  been  too  busy  “doing”  to  develop  lessons  learned 

•  But  need  to  do  more  pilots  first 


—  Software  Engineering  Institute  Carnegie  Mellon 


D2007  Carnegie  Mellon  University 
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For  Additional  Information 


Jay  Douglass 
Business  Development 
Product  Line  Systems  Program 
Telephone:  412-268-6834 
Email:  jcd@sei.cmu.edu 


Linda  Northrop 
Director 

Product  Line  Systems  Program 
Telephone:  412-268-7638 
Email:  lmn@sei.cmu.edu 


Technical  Details: 

Mike  Gagliardi 

Product  Line  Systems  Program 
Telephone:  412-268-7738 
Email:  mjg@sei.cmu.edu 

World  Wide  Web: 
http://www.sei.cmu.edu/architecture 


U.S.  Mail: 

Software  Engineering  Institute 
Carnegie  Mellon  University 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213 

SEI  Fax:  412-268-5758 


Software  Engineering  Institute 


Carnegie  Mellon 


©2007  Carnegie  Mellon  University 
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DoD  Software  Engineering  and 
System  Assurance 


Kristen  Baldwin 

Deputy  Director,  Software  Engineering  and 

System  Assurance 

Office  of  the  Under  Secretary  of  Defense 
Acquisition,  Technology  and  Logistics 


Elements  of  AT&L  Strategy  for 

Software 


•  Support  Acquisition  Success 

-  Ensure  effective  and  efficient  software  solutions  across  the 
acquisition  spectrum  of  systems,  SoS  and  capability  portfolios 

•  Improve  the  State-of-the-Practice  of  Software 

Engineering 

-  Advocate  and  lead  software  initiatives  to  improve  the  state-of- 
the-practices  through  transition  of  tools,  techniques,  etc. 

•  Leadership,  Outreach  and  Advocacy 

-  Implement  at  Department  and  National  levels,  a  strategic  plan 
for  meeting  Defense  software  requirements 

•  Foster  Software  Resources  to  meet  DoD  needs 

-  Enable  the  US  and  global  capability  to  meet  Department 
software  needs,  in  an  assured  and  responsive  manner 


Promote  World-Class  Leadership  for  Defense  Software  Engineering 


Top  Software  Issues* 


1 .  The  impact  of  requirements  upon  software  is  not  consistently  quantified  and 
managed  in  development  or  sustainment.  “Requirements” 


2.  Fundamental  system  engineering  decisions  are  made  without  full 
participation  of  software  engineering.  “SE/SW  Integration” 

3.  Software  life-cycle  planning  and  management  by  acquirers  and  suppliers  is 
ineffective.  “SW  Sustainment” 

4.  The  quantity  and  quality  of  software  engineering  expertise  is  insufficient  to 
meet  the  demands  of  government  and  the  defense  industry. 

“Human  Capital” 

5.  Traditional  software  verification  techniques  are  costly  and  ineffective  for 
dealing  with  the  scale  and  complexity  of  modern  systems.  “SW  Testing” 

6.  There  is  a  failure  to  assure  correct,  predictable,  safe,  secure  execution  of 
complex  software  in  distributed  environments.  “SW  Assurance” 

7.  Inadequate  attention  is  given  to  total  lifecycle  issues  for  COTS/NDI  impacts 
on  lifecycle  cost  and  risk.  “SW  COTS/NDI/Reuse” 


*NDIA  Top  Software  Issues  Workshop 
August  2006 


OSD  Software  Systemic  Analysis 


•  OSD(AT&L)/SSE  Systemic  Analysis  Database 

•  Current  Dataset:  68  reviews  on  38  different  ACAT  1 D 
systems  acquisition  programs  since  early  2004 

-  Approx  4,000  findings  from  these  reviews  placed  into  formal 
database  repository 

•  Data  extracted  using  the  following  key  words: 

-  Software 

-  Systems-of-Systems  (SoS) 

-  Assurance 

-  Architecture 

-  Security 

•  600+  findings  resulted  from  the  keyword  search 
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Data  Validation 


Data  validation  was 
conducted  to: 


Software  Related  Findings 
Total:  284 


Remove  any  extraneous 
records  from  the  resulting 
report  unrelated  to  SW 

Ensure  that  positive,  neutral, 
and  negative  findings  were 
identified  properly 


•  Resulted  in  284  Directly  □  Positive  ■  Neutral  □  Negative 

Software  Related  Findings 


What  leads  to  Software  Problems 

in  DoD  Programs? 


I 


S  ^ 


Tier  1  Trends  -  Level  1 

Tier  1  Trends  -  Level  2 
(Derivative  of  Tier  I  Trend 


Last  Updated:  August  2008 


Human  Capital 

Insufficient  availability  of 
qualified  software 
engineering  personnel 
with  necessary  skills  and 
expertise 


There  is  inadequate 
sharing  of  knowledge 
related  to  software 
engineering  issues,  risks, 
and  lessons  learned 
within  and  across 
programs  and  services 
Knowledge  Sharing 


Management  Oversight 

There  is  a  failure  to  establish 
program-wide  governance  for  all 
software  engineering  activities 


Management  Oversig 

Program  software  engineering 
status  is  inadequately  tracked  V 
against  plans  throughout 
programs"  lifecycles 


Management  Oversight 

There  is  an  underestimation 
of  the  complexity  of  software 
integration  efforts 


Software 
problems 
inherent  in 
DoD  Programs 


Misapplied  software  engineering 
processes  are  adversely  impacting 
management  oversight 

Process  Planning 


There  is  a  lack  of  emphasis 
on  software  architecture  quality 
attributes  and  priorities  in  Software 
requirements  documents 
Architecture 
There  are  inadequate 
software  architecture  designs 
Architecture 


Detailed  Results  of  Overarching  Trends 


o 

> 

<D 


Knowledge  Sharing 

i — i 

Human  Capital 

There  is  inadequate  sharing  of 

1 _ 1 

Insufficient  availability  of  qualified 

knowledge  related  to  software 

□ 

software  engineering  personnel  with 

engineering  issues,  risks,  and  lessons 

necessary  skills  and  expertise 

learned  within  and  across  programs 

and  services 

□ 

I 


Tier  1  Trends  -  Level  1 


f 


i 


Management  Oversight 

There  is  a  failure  to  establish 
program-wide  governance 
for  all  software  engineering 
activities 


◄- 


-► 


Management  Oversight 

There  is  an  underestimation 
of  the  complexity  of  software 
integration  efforts 


◄- 


-► 


Process  Planning 

Misapplied  software 
engineering  processes  are 
adversely  impacting 
management  oversight 


Schedule  Estimation 

Poor  communication  of  schedule 
status 


Sustainment  /  Maintenance 

Inadequate  planning  of  software 
sustainment/maintenance  activities 


-► 


SW  COTS/Reuse 

Poor  software  estimation  analysis  for  COTS/ 
reuse  within  the  program 


CM 

0 

> 

<D 
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Software  Testing 

Inconsistent  Test  Process  Management 
-planning 


Systems  and  Software  Integration 

Lack  of  engineering  plans  for  integration  such 
as  CONOPS  and  architecture 


Management  Oversight 

Program  software  engineering  status  is 
inadequately  tracked  against  plans 
throughout  programs"  lifecycles 


Software  Metrics 

Lack  of  clear  insight  into  status  of 
software  activities  throughout 
program  lifecycle 

Software  Metrics 

Inability  to  maintain  accountability 
during  program  lifecycle 


Resource  Allocation 

Underestimation  of  available  budget  and 
resources 


-► 


Software  Configuration 
Management 

Lack  of  emphasis  on  configuration 
management  process 


Schedule  Estimation 

Lack  of  detail  in  planning  leading 
to  schedule  delays 


EVM 

Over  reliance  on  EVM  to  provide 
visibility  into  schedule  risks 


Systems  and  Software  Integration 

Lack  of  authority  to  manage  integration  of 
systems  (i.e.  Multi-platform,  legacy  systems) 


Risk  Management 

Software  complexity  (GFE/COTS), 
requirements  instability,  and  time  constraints 
contribute  to  inadequate  risk  identification  and 
management  (i.e.  updating  of  legacy  systems) 


£ 


Architecture 

There  are  inadequate 
software  architecture 
_ designs _ 


Software  Assurance 

Lack  of  software  assurance  guidelines. 
Evident  in  lack  of  coordination  across  security 
plans/processes,  unclear  countermeasure 
efforts/techniques,  lack  of  understanding  of 
foreign  involvement  standards 


Architecture 

There  is  a  lack  of  emphasis  on 
software  architecture  quality 
attributes  and  priorities  in 
software  requirements 
documents 


Requirements  Engineering 

Requirements  gathering  is 
incomplete  (i.e.,  lack  of 
funding,  over  reliance  on 
contractor,  staff  experience, 
and  immature  technology) 


I 


Requirements  Management 

Inadequate  Requirements 
Management  process  causing 
undeveloped  definition  of 
requirements  and  lack  of 
traceability 


NDI  A/DUS  D(A&T)SSE 
Issues  Validation 


National  Defense  Industrial  Association  (NDIA)  DUSD(A&T)  SSE  Directorate 

Top  7  Software  Issues  Program  Review  Software  Systemic  Analysis  Findings 

August  2006 


Software  Human  Capital 


Software  Requirements 


Systems/Software 

Integration 


Software  Human  Capital 

■  Resources 

■  Quality  Level 

Software  Requirements 

■  Engineering 

■  Management 

■  Acquisition  Strategy 

Systems/Software 

Integration 

■  Systems  of  Systems 

■  Interoperability 

■  Tech  Refresh 

Software  Assurance 


Software  Testing 


Software  Sustainment 


Software  COTS/NDI 


Software  Assurance 


Software  Development 

■  Software  Testing* 

■  Software  Sustainment/Maintenance* 

■  Software  COTS/NDI* 

■  Technology  Readiness 

■  Software  Architecture 

Software  Metrics 

■  Software  Metrics 

■  EVM 

Software  Engineering 
Management 

■  Project  Planning 

■  Management  Oversight 

■  Software  Configuration  Management 

Knowledge  Sharing 

■  Process 

■  Reporting 

SW  Roundtable  Results 


•  Shared  Army,  Navy,  Air  Force  software  strategies 

-  Found  synergy  in  many  areas 


•  Identified/prioritized  22  proposed  initiatives  to  tackle 
software  issues  -  Top  5  of  these: 

-  Synergize/Harmonize  "core  SW  metrics”  across  DoD;  develop 
approaches  for  incorporating  them  into  gate  reviews,  processes, 
earned  value 

-  Organize  start-up  teams  and  infrastructure  to  facilitate  software 
program  success 

-  Establish  SE/SW  architecture  “review  board”  to  engage  early 
with  programs  and  provide  constructive  suggestions 

-  Define  analysis  process  for  reuse/reusable  assets  to  improve 
estimation  accuracy:  including  consideration  of  product  features 

-  Develop  approaches  for  SW  testing  and  evaluation  to  enable 
mission  success 


ODUSD(A&T)  SSE/SSA  Way 

Forward 


•  Goal:  Prosecute  top  software  and  assurance  issues 


•  SSA  FY08/09  Activities: 

-  SW  Lifecycle  Touchpoints:  SW  guidance  to  complement 
Enhanced  SE  and  SE  Technical  Reviews 

-  SW  Human  Capital  Strategy:  Graduate-level  and  DoD 
acquisition  workforce  software  curricula 

-  SE/SW  Integration:  Design  a  framework  to  define  and 
measure  integration.  Partnership  with  academia,  industry 

-  SW  Measurement:  Guidance  on  collection  and  use  of  SW 
Data 

-  SW  Test,  SW  Reliability:  New  in  FY09 

-  System  Assurance:  SA  Guidebook;  Program  Protection 
Policy/Guidance,  DIB  Cyber  Security  Strategy 


DoD  SW  Community  Way  Forward 


•  Review  all  initiatives  to  determine  opportunity  for 
collaboration/augmentation 

-  DoD  Software  Working  Group 

-  NDIA  Software  Expert  Panel 

•  Discuss  plans  for  individual  initiatives  (top  5)  on 
Collaborator  teleconferences 

•  Organize  collaborator  events  for  FY09 

-  Focused  working  groups/workshops  as  appropriate 

•  Continue  to  increase  software  visibility  in  NDIA 
SE  Conference 

-  Plan  event  for  FY09 


•  Threats:  Nation-state,  terrorist,  criminal,  rogue  developer 
who: 

-  Gain  control  of  IT/NSS/Weapons  through  supply  chain  opportunities 

-  Exploit  vulnerabilities  remotely 

•  Vulnerabilities:  All  IT/NSS/Weapons  (incl.  systems, 
networks,  applications) 

-  Intentionally  implanted  logic  (e.g.,  back  doors,  logic  bombs,  spyware) 

-  Unintentional  vulnerabilities  maliciously  exploited  (e.g.,  poor  quality 
or  fragile  code) 

•  Consequences:  Stolen  critical  data  &  technology;  corruption, 
denial  of  critical  warfighting  functionality 

System  Assurance  is  the  confidence  that  the  system 
functions  as  intended  and  is  free  of  exploitable  vulnerabilities, 
either  intentionally  or  unintentionally  designed  or  inserted 

during  the  lifecycle 
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Program  Protection  -  The  Road  Ahead 


•  DoD  System  Assurance 

-  Evolved  from  Software  Assurance  Efforts 

-  Creates  a  ..framework"  to  integrate  multiple  security  disciplines 
and  policies 

-  Leverages  5200.39:  expanding  CPI  definition  to  include  system 
assurance  and  total  life  cycle 

•  DoDI  5200.39  CPI:  Three  Categories  of  CPI: 

-  Information,  Technology,  Components 

•  Programs  will 

-  Define  CPI  at  Milestone  A 

-  Develop  a  Program  Protection  Plan  (PPP)  for  Milestone  B 

-  Be  Subject  to  Review  and  Oversight 

-  Execute  mitigation  strategies  (such  as  use  of  Trusted  Foundries 
or  Anti-Tamper) 


Engineering  for  System  Assurance 


•  “Engineering  for  System  Assurance”  VI  .0  Guidebook 
signed  out  at  NDIA  October  1 , 2008 

•  Posted  on  SSE  Web  site  at: 

-  http://www.aca.osd.mil/sse/ssa/auidance.html 

•  Provides  guidance  on  how  to  address  System 
Assurance  through  Systems  Engineering  processes 

-  Aligns  to  DoD  acquisition  lifecycle  processes  with  actionable 
criteria 

-  Adds  emphasis  to  ISO/IEC  15288  SE  processes 

•  Enhanced  IA  focus  and  alignment  with  current  processes 

-  Focus  on  hardware,  software  and  operational  environment 

-  Dovetails  with  Program  Protection  Planning  (PPP)  processes 

-  Supports  identification  of  trusted  foundry  resources 

-  Informs  Anti-tamper  considerations 


Expanding  DoD  Industry  Partnership 


•  Acquisition  Cyber  Security  is  a  long  term  interest  for  DoD 

-  Fully  anticipating  Cyber  Security  is  expected  to  be  a  ongoing 
priority  for  the  new  administration 

•  DoD  will  continue  to  take  advantage  of  the  global 

marketplace  and  COTS  solutions 

-  Engineering  for  System  Assurance  seeks  to  identify  and  fortify 
critical  components  allowing 

•  Industry  is  part  of  the  solution 

-  NDIA  System  Assurance  Committee  will  continue  to  focus  on  the 
solution  strategy 

-  ITAA,  GEIA,  INCOSE,  others  all  participate  on  this  committee 


Questions 
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Implications  of  Capability-based  Planning 
on  Requirements  Engineering 


Leonard  Sadauskas 

Presented  at 
NDIA  SE  Conference 

Requirements  Development  &  Management  Session 

22  October  2008 


Disclaimer 


The  views  and  opinions  expressed  in  this  presentation  are 
those  of  the  author  and  do  not  reflect  the  policy  of  the 
Department  of  Defense 


22  Oct  2008 


Leonard  Sadauskas 
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Scope  of  this  Presentation 


•  Capability-based  planning 

•  The  problem  and  solution  space  interface 

•  The  dual  roles  of  measures  of  effectiveness  (MOEs) 

•  Capability  feedback  process 

•  Issues,  challenges  and  trends 


22  Oct  2008 
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Definitions 


•  Capability-based  planning  (CBP): 

-  An  overarching  framework  for  planning  under  uncertainty 
that  provides  capabilities  suitable  for  a  wide  range  of 
modern-day  challenges  and  circumstances  while  working 
within  an  economic  framework  that  necessitates  choice 

•  Capabilities-Based  Assessment  (CBA) 

-  Study  that  identifies  the  capabilities  (and  operational 
performance  criteria)  required  to  successfully  execute 
missions 

•  Capability: 

-  The  ability  to  execute  a  specified  course  of  action 

•  Move  troops  rapidly  _ 


Candidate  Solutions: 

•  Truck 

•  Ship 

•  Aircraft 
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Capabilities-based  Planning  Framework 
(work  in  progress  since  2003) 


Feedback  ) 

(CBA) 

Fielded 

—  — j 

^ '  CBP 

Capabilities 

/  Non-materiel 

Analysis 

\  Solutions 

N 

V  J 

V  J 

Acquisition 


PPBE 
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Focus  of  this  Presentation 


Problem  Space 

Solution  Space 

Fielded  Capability 

CBA 

Acquisition 

O&S 
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Draft  CJCSI  31 70.01  G  JCIDS  Process 
and  Acquisition  Decisions 


Problem 

Space 


Solution 

Space 
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Capabilities-Based  Assessment  (CBA) 


Existing 

Guidance 


i 


FAA 


What  are 
we  talking 
About? 


i 


FNA 


From: 

CBA  User’s  Guide 
V2  December  2006 


How  good 
Are  we  at 
Doing  it? 

i 

FSA 


What 

Should  we 
Do  about  it? 
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Draft  5000.02  The  Defense  Acquisition 
Management  Framework. 


User  Needs 


Technology  Opportunities  &  Resources 


•  The  Materiel  Development  Decision  precedes 
entry  into  any  phase  of  the  acquisition  framework 

•  Entrance  criteria  met  before  entering  phase 

•  Evolutionary  Acquisition  or  Single  Step  to 
Full  Capability 


A  A1 


(Program 

Initiation) 


A 


IOC 


FOC 


Materiel 

Solution 

Analysis 

Technology 

Development 

Engineering  and 
Manufacturing 
Development  & 
Demonstration 

Production  & 
Deployment 

Operations  & 
Support 

^  Materiel 
>  Development 
y  Decision 

/\Post-CDR 
\/  Assessment 

LRIP/IOT&E  A  Decision 
v  Review 

Pre-Systems  Acquisition/  \ 


/\  Sustainment  / 


Systems  Acquisition 


-  Decision  Point  /\=  Milestone  Review 
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Problem  /  Solution  Space  Interface 


Requirements  /  Systems 

Engineer 


Business  Analyst 
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Information  Transfer  at  the  Interface 


If  your  goal  in  software  development  is  to  "make  the  business  case 
come  true"  (and  by  'business  case'  I  mean  the  initial  justification  for 
spending  time,  money,  and  effort  on  the  development  in  the  first 
place),  then  the  most  important  thing  to  understand  is:  why  are  we 
building  this?  That  is,  what  are  the  needs  of  the  customers  (or 

business)?  If  you  don't  know,  or  clearly  understand,  the 

customer  needs,  then  you  cannot  know  if  you  are 

building  the  right  system  -  which  then  makes  the  technical 
correctness  of  the  functional  spec  (what  we  intend  to  build)  or  the 
design  spec  (how  we  think  it  should  work)  a  moot  point. 

Richard  Zultner 

30  Sep  2008  Requirements-Engineering  Group 
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AoA  and  Effectiveness  Analysis  Process 
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Definitions  and  Attributes  of  MOEs 


MOEs  are  standards  against  which  the  capability  of  a 
solution  to  meet  the  needs  of  a  problem  may  be  judged. 
The  standards  are  specific  properties  that  any  potential 
solution  must  exhibit  to  some  extent. 

Therefore,  MOEs  are  independent  of  any  solution. 

A  meaningful  MOE  must  be  quantifiable  and  a  measure 
to  what  degree  the  real  objective  is  achieved. 


The  MOE  is  part  of  both  the  AoA  and 
the  CBP  feedback  process 
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Feedback  Process 


Multiple  sources  of  capability  information 
Separate  JS,  COCOM  and  Service  Processes 
Not  part  of  JCIDS  or  AMS 
Statutory  for  fielded  capability  as 

Post  Implementation  Review  (PIR)  _ 


77\ 

PIR 

Fielded  Capability 
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Post  Implementation  Review  (PIR) 

Defined 


An  analysis  of  an  investment  or  acquired  system 
that  is  part  of  a  capability  portfolio,  operating  in 
its  intended  environment,  using  data  collected 
from  various  sources  to  answer  the  question: 


Did  we  get  what  we  needed,  and  if  not  what  to  do 
about  it? 
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PIR  Information  Path  in  Feedback  Process 
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PIR  in  the  Feedback  Process 


FCB/Sponsor 


MOEs 


ICD 


PIR 


Contract 


Build 


Integration 
&  Test 


Platform  Readiness  Assessments 
CC  Exercise  results 
User  Satisfaction  Surveys 
Annual  CFO  Report  Input 
Mission  Readiness  Assessments 
ROI  Computation 
War  Games 


MS  A  MSB  MS  C  IOC 


PIR 


FOC 
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Opportunities,  Challenges  and  Trends 
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Model  Compatibility  &  Sharing  Opportunity  at  the 

Problem-Solution  Interface 


Mission 

Tasks 


Functional 


Needs  ^ 


MOEs 


Initial  Capabilities  Document 
(ICD) 


Materiel 
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Study  Operational  Effectiveness  Analysis 
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Alternative 
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Alternative 
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MOE  Deficiencies  in  CBA 


25 

20 

15 

10 
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Dec  2005  through  Jul  2008 
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Potential  Impact  of  MOE  Deficiencies 


Likely  Scenario: 

-  43%  ICDs  submitted  to  the  JS  for  review  during  past  30 
month  period  contained  no  MOEs 

Assumptions  (conservatively  stated) 

-  Requirements  volatility  accounts  for  10%  of  Program  of 
Record  cost  overruns. 

-  Lack  of  MOEs  accounts  for  10%  of  requirements  volatility 

-  The  2008  DoD  Major  Program  cumulative  expenditure  is 
$800B  +  $800B  less  than  major  =  $1 ,600B 

-  Cost  overrun  is  5%  or  $80B 

Cost  of  not  providing  MOEs  to  the  SE  process: 

-  .1  x.l  x  .43  x  $80B  =  $344M 
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Recent  Trends 


•  Publication  of  CBA  Guide  v2  by  JS-J8  in  Dec  2006 

-  Describes  CBA  process 

-  Guidance  for  study  plan  and  planning 

-  Discusses  analytic  approaches 

•  Development  of  MOEs 

•  Implementation  of  requirements  manager  training  and 
certification 

-  USD(AT&L)  Memoranda  of  2  September  2008,  Requirements 
Management  Certification  Training  Program  Policy,  John  Young 

-  Includes  training  and  certification  of  requirements  authors, 
reviewers  and  validators 

•  Joint  Staff  considering  shortening  the  CBA  cycle  to  a  month  or 
two  instead  of  a  year  or  two. 

-  Impact  on  development  of  MOEs  not  clear 

-  May  be  signal  that  Problem-Solution  interface  boundary  is  shifting 
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Some  Remaining  MOE  Issues 

•  How  could  MOEs  be  allocated? 

^^^DesiredCapability  —  MOE  Existing  Capability  MOE  Gap  Capability 

MOEGap  capability  =  MOEdotlpf+D  MOE,CDs 
where  DOTLPF  =  /(Existing  processes  +  changes  needed 

to  maximize  benefit  of  materiel  investment) 

•  How  could  MOEs  be  traced? 

Could  MOEs  be  traced  through  the  DOTLPF  and  materiel  acquisition 
processes  in  a  manner  analogous  to  requirements  tracing  by  the 

systems  engineers? 
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PIR  Input  to  JCA  Assessment 
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Provide  Operational  Capability 
with  fielded  Systems  to  meet 
Warfighter  requirements 
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warfighting 
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Models  Bridge  Layers  of  Requirements 
and  Provide  Verification  Criteria 
INCOSE  Work  Shop  08 


22  Oct  2008  Leonard  Sadauskas  26 

After  Jeremy  Dick’s  Sandwich  Requirements  &  Modeling  Concept 


Typical  MOE  Situations 


•  Outcome  metrics  presented  but  measures  deferred  for  ODD 

•  Study  team  not  adequately  staffed 

•  Study  team  neither  tasked  nor  funded  to  undertake  analytic 
approach  needed  to  develop  MOEs 

•  Outcome  measures  stated  in  narrative  but  solution  performance 
parameters  KPPs  presented  as  MOEs 

•  CJCSM  3170.01  does  not  explicitly  require  MOEs  for  the  ICD, 
Draft  CJCSI  31 70.01  G  has  eliminated  the  term  MOE 

-  Uses  the  term  desired  effects 

•  Developed  MOEs  do  not  address  desired  outcomes 
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Cause  -  Effect  Candidates 


Lack  of  capability  analyst  training 

-  Analyst  jumps  into  solution  mode  comfort  zone 

Capability  lexicon  confusion 

-  Miscommunication  amongst  analysts  and  reviewers 

Regulatory  MOE  requirement  inconsistencies 

-  Analyst  takes  path  of  least  work 

-  ICD  approval  available  without  MOEs 

Inadequate  study  team  guidance 

-  Analyst  not  steered  to  analytic  approaches  needed  to  develop  MOEs 
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►  Rationale:  Why  integrate  systems  and  software 
engineering? 

►  Touchpoint:  A  framework 

►  Initial  Results 

►  Next  steps 


Rationale:  Assertions 


►  Interdependent  systems  are  those  where: 

►  A  "major"  portion  of  the  capabilities/value  of  the  system  is 
delivered  through  software 

►  A  "major"  portion  of  system  quality  attributes  "largely" 
depend  on  software  (safety,  security,  agility,  reliability, 
availability,  resilience,...) 

►  Today  most  high  value  systems  are  interdependent; 
that  percentage  is  increasing 

►  In  these  systems,  nearly  all  important  decisions 
require  equal  consideration  of  software 
engineering  and  systems  engineering  expertise 

►  Technical,  management,  personnel  and  customer 
concerns  are  included 

►  But,  what  does  it  mean  to  integrate  SE  and  SwE? 
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Rationale:  Questions  needing  answers 


1.  What  outcomes  do  we  expect  from  SE/SwE 
integration? 

►  Does  integration  reduce  key  risks? 

2.  How  do  you  measure  integration  or  it’s 
outcomes? 

3.  How  and  why  do  the  SwE  and  SE  activities 
conflict,  complicate,  or  reinforce  each  other? 

4.  How  much  integration  is  needed? 

►  What  is  the  scope  of  integration  (development, 
operations,  business  areas...)? 

►  Is  more  integration  always  better? 

►  Is  integration  domain-  or  application-dependent? 

5.  Why  haven’t  IPTs  or  CMMI  solved  this  problem? 
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Rationale:  Barriers  to  integration 

►  Historical  context  and  vestigial  prejudices 

►  SE  and  SwE  cultures  are  significantly  different 

►  SE  and  SwE  have  different  educational  backgrounds 

►  SE  and  SwE  vocabularies  are  similar  but  meanings 
differ 

►  SE  and  SwE  process  implementations  are  often 
incompatible  (e.g.  V  versus  spiral) 

►  SE  and  SwE  may  use  the  same  tools  differently 
(UML) 

►  No  language  to  discuss  integration  of  SE  and  SwE 


Rationale:  Issues  needing  to  be  addressed 


1.  Vocabulary.  There  is  no  precise  way  to  talk 
about  the  integration  of  systems  and  software 
engineering. 

2.  Measurement.  There  is  no  precise  way  to  talk 
about  how  much  integration  there  is  between 
systems  and  software  engineering  in  a  particular 
situation. 

3.  Entanglement.  The  complexity  of  the  disciplines 
makes  it  difficult  to  identify  where  software  and 
systems  engineering  touch. 

4.  Value.  There  is  no  comprehensive  list  of  benefits 
that  can  be  achieved  by  integrating  systems 
and  software  engineering  nor  is  there  an 
understanding  of  the  associated  costs. 
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Touchpoint 

►  A  framework  to  support  the  discussion  of  SE/SwE 
integration 

►  Simple  and  (seemingly)  robust 

►  Provides  a  way  to  describe  integration  at  the 
practitioner  level 

►  Describes  touchpoints  where  the  two  disciplines 
interact 

►  May  help  to  describe  the  degree  of 
“integratedness” 


Touchpoint  Framework:  Components 

►  Processes.  The  ordered  activities  that  define  the 
systems  and  software  engineering  disciplines 

►  Touchpoints  (TPs).  The  two  discipline’s  processes 
touch  when  interactions  between  their 
constituent  activities  affect  program  risk  or  value 
-  positively  or  negatively. 

►  Faults.  A  touchpoint  may  exist,  but  the  process  or 
activity  may  fail  to  produce  its  maximum  value. 

►  Resolution  Strategies  (RSs).  For  each  fault,  there 
may  be  one  or  more  actions  that  will  eliminate 
the  fault  or  reduce  its  impact. 
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Touchpoint  Framework:  Processes 


►  ISO  15288  provides  “harmonized”  systems  and 
software  engineering  processes 

►  Agreement,  Organizational  Project-enabling, 
Project,  and  Technical  processes 
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Touchpoint  Framework:  Faults 


►  Gap 

►  Logically,  there  should  be  an  interaction  between  the 
corresponding  SE  and  SwE  processes,  but  the 
processes  do  not  include  one.  A  needed  activity  is 
therefore  performed  poorly,  or  not  performed  at  all. 

►  Clash 

►  One  or  more  activities  in  each  of  the  two 
corresponding  SE  and  SwE  processes  produce  are 
incompatible  and  result  in  inconsistent  results  or 
inconsistent  actions. 

►  Waste 

►  Activities  in  the  two  corresponding  SE  and  SwE 
processes  independently  expend  resources  that 
produce  the  same  result  or  take  the  same  action  with 
no  added  benefit  to  the  program 
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Touchpoint  Framework:  Faults  -  Clashes 


►  Vocabulary 

►  SE/SW  activities  use  the  same  terminology  with 
different  meanings,  or  terms  not  recognized  by  the 
other,  making  communication  harder 

Example:  Object-oriented  terminology 

►  Value 

►  Software  and  systems  engineers  in  an  organization  or 
program  value  different  process  characteristics 

►  Example:  Stability  of  baselines 

►  Mental  Model 

►  Software  and  systems  engineers  think  differently  about 
how  to  carry  out  process  activities 

►  Example:  “part-of”  relationships  vs.  “uses”  relationships. 
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Touchpoint  Framework:  Example  TP 


Process 

Touchpoint 

Fault 

Type 

Architectural 

Design 

Systems  architectures 
include  significant 
software  components 
to  deliver  critical 
capability 

Software-engineering 
architectures  define  layers  of 
related  functionality,  while  most 
systems-engineering  methods  are 
hierarchical  structures. 

Clash  - 
Mental  Model 

Example  from  pilot  research 
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Touchpoint  Framework:  Resolution  Strategies 


►  There  is  a  desire  to  fix  faults,  especially  those  with  high 
impact  on  risk  or  value. 

►  For  each  fault,  there  may  be  one  or  more  resolution 
strategies,  which,  when  executed  well,  will  eliminate 
the  fault  or  at  least  reduce  its  impact. 

►  In  some  cases,  resolution  strategies  are  known  and  just 
need  to  be  applied 

►  On  the  other  hand,  resolving  some  faults  will  require 
research 

►  Resolution  strategies  are  grouped  into  four  traditional 
categories:  process ,  people ,  environment,  and 
technology.  Any  number  of  resolution  strategies  in 
each  category  is  possible  for  a  fault. 
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Touchpoint  Framework:  Example  RSs 


Process 

Touchpoint 

Fault 

Type 

Architectural 

Design 

Systems  architectures 
include  significant 
software  components 
to  deliver  critical 
capability 

Software-engineering  architectures 
define  layers  of  related  functionality, 
while  most  systems-engineering 
methods  are  hierarchical  structures. 

Clash  - 
Mental 

Model 

Resolution  Strategy 

Category 

Research  must  be  conducted  to  resolve  the  clash  between  object- 
oriented  and  structured  methods.  Maier  provides  some  of  the  best 
research  in  this  area. 

Technology 

Design  software  architecture  to  look  just  like  system  architecture.  Make  it 
easy  for  a  system  architect  to  understand  (SW  systems  mirror  HW  systems , 
e.g.  relays,  motors,  etc).  Then  SW  helps  the  system  architect  understand 
things  in  better  detail. 

Process 

Middleware  may  be  able  to  bridge  the  gap. 

Technology 

Examples  from  pilot  research 
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Touchpoint  Framework:  Measurement 

►  Provides  a  way  to  measure  how  much  integration  has 
been  achieved  and  how  good  that  integration  is. 

►  The  amount  of  integration  is  simply  the  total  number 
of  touchpoints  in  the  implementation  of  the  25 
processes  -  a  higher  number  indicates  more 
integration. 

►  A  somewhat  more  sophisticated  approach  associates  a 
weight  with  each  touchpoint  to  reflect  its  potential  impact 
on  program  risk  or  value. 

►  The  number  of  faults  determines  integration  quality. 

►  Faults  can  also  be  weighted  based  on  their  consequence. 

►  A  fault  that  severely  impacts  an  important  touchpoint 
would  be  of  far  greater  consequence  than  a  fault 
that  barely  impacts  a  minor  touchpoint. 
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Initial  research:  Piloting 


►  Process  activities  at  the  “touchpoint”  level  are 
generally  not  found  in  available  traditional 
documentation  (standard  processes,  WBS,  plans) 

►  Often  technical  management/practitioner  activities 

►  Approach  -  interview  SE  and  SwE  leadership 

►  Identified  ~10  programs  through  OSD  AT&L  and  NDIA 

►  Interviewed  each  program  to  identify  touchpoints, 
faults,  resolution  strategies  and  challenges;  rigid  “no 
attribution”  policy 

►  Compared  interview  findings  with  the  systemic 
analysis  findings  of  AT&L/SSE  Program  Support 
Assessments 
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Piloting  Results 


►  Touchpoint  elements  (TPs,  Faults,  RSs)  identified  by 
Systemic  Analysis  Category 


Category 

Elements 

No.  of  Projects 

Architecture 

12 

6 

CM 

1 

1 

EVM 

2 

2 

Human  Capital 

4 

2 

Process  Planning 

3 

3 

Requirements 

23 

10 

Risk  Management 

2 

2 

System  Integration 

4 

4 

Software  Metrics  (Visibility) 

4 

3 
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Piloting  Results 


►  Touchpoint  elements  not  in  Systemic  Analysis 
Category 


Category 

Elements 

No.  of  Projects 

Contracting 

4 

3 

Life  Cycle 

7 

4 

Technical  Reviews 

2 

2 
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Sample  Architectural  Design  Process  Findings 


Touchpoint 

Fault 

Type 

Architecture  concept 

Underutilized  software  capability 

Gap 

Resolution  Strategy 

Category 

Concept  development  should  be  performed  jointly  and  careful  trades 
made  that  reflect  HW  and  SW  capabilities,  strengths,  and  weaknesses 

Process 

Touchpoint 

Fault 

Type 

Meeting  non-functional 
requirements 

HW  reliability  numbers  are  calculated  to 
many  decimal  places,  and  include  the 
contributions  of  very  low-level  WBS 
components.  SW  reliability  is  not 
understood  and  so  ignored. 

Gap 

Resolution  Strategy 

Category 

Research  in  integrated  reliability  approaches  is  needed 

Technology 

Train  systems  and  reliability  engineers  to  understand  software  reliability 

People 

From  pilot  research 
A  uthors  ’  suggestion 
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Sample  Requirements  Analysis  Process  Findings 


Touchpoint 

Fault 

Type 

Software  Requirements 

SW  specifications  that  limit  trade  space 

Clash  - 
Mental 
Model 

Resolution  Strategy 

Category 

Define  software  requirements  in  terms  of  “what”  not  “how.” 

Process 

SE  and  SW  collaborate  in  the  development  of  software  requirements 

Process 

Touchpoint 

Fault 

Type 

Requirement  Maturation 

The  difference  in  speed  of  maturation 
between  HW  and  SW  requirements  causes 
tension  between  SEs  and  SwEs. 

Clash  - 
Mental 
Model 

Resolution  Strategy 

Category 

Requirements  management  tools  and  processes  need  to  better  support  iterative 
approaches  to  requirements  maturation. 

Technology 

From  pilot  research 
Authors’  suggestion 
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Sample  Life  Cycle  Management  Process  Finding 


Touchpoint 

Fault 

Type 

SE  and  SW  life  cycles 

Life  cycle  speeds  differ  causing  perceived 
architecture  instability  and  schedule 
coordination  problems 

Clash  -Value 

Resolution  Strategy 

Category 

Involve  SEs  in  software  projects  using  iterative  life  cycles  to  gain  comfort  and 
trust 

People 

From  pilot  research 
Authors’  suggestion 
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Conclusions  and  Next  steps 

►  Framework  seems  useful 

►  Need  much  more  data 

►  More  programs 

►  More  variety 

►  Refine  and  extend  initial  findings  with  new  data 

►  Create  products  that  make  findings  useful  to 
programs 
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Questions  and  Discussion 
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Presentation  Outline 


■  Threat  Reduction  Analytic  Objectives 

■  Original  Long-Term  Vision  for  the  Constructive 
Simulation  Environment 

■  Overview  of  Spiral  Development  /  Analysis  Approach 

■  Scenario  Vignettes 

■  Nuclear  Radiation  Detection  Modeling 

■  Behavior  Module  Characteristics 

■  Near-Term  Plans 

The  work  presented  herein  was  supported  by  the  Defense  Threat  Reduction  Agency  (DTRA) 
under  NAVSEA  Contract  N00024-03-D-6606,  Task  SG412. 


Threat  Reduction  Analytic  Objectives 

Issues 

"  Many  potential  system  solutions  are  being  proposed  for  detection  of 
materials  important  to  Weapons  of  Mass  Destruction  (WMDs) 

■  System  effectiveness  evaluation  requires  analysis  of  system 
performance  in  operationally  realistic  tactical  vignettes 

■  Material  detection  effectiveness  is  an  element  of  broader  campaign-level 
scenarios 


Overall  analysis  objective: 

■  To  enable  system  acquisition  decision-making  by  constructing  an 
analysis-of-alternatives  capability 


Immediate  analysis  objective 

■  Construct  an  analysis-of-alternatives  capability  focused  on  detection  of 
nuclear  materials 


Original  Long-Term  Vision  for  the 
Constructive  Simulation  Environment 
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Overview  of  Spiral  Development  /  Analysis 
Approach 

■  Spiral  0  (April  -  July  2007) 

■  Proof-of-concept  use  of  Joint  Semi-Automated  Forces  (JSAF) 
simulation  for  radiation  detection  in  tactical  vignette 

■  Spiral  1  (August  2007  -  January  2008) 

■  Setup  of  M&S  laboratories  at  DTRA  and  JHU/APL 

■  Development  of  checkpoint  scenario  vignettes 

■  Development  of  higher  fidelity  JSAF  radiation  detection  module 
and  production  of  initial  passive  detection  sensor  performance 

■  Spiral  2  (February  -  September  2008) 

■  Expansion  of  scenario  vignettes,  with  3D  rendering 

■  Development  of  Behavior  Module  linked  with  JSAF 

■  Initial  development  of  JSAF-embedded  software  for  active 
concepts  for  nuclear  material  detection 


Spiral  0  Activities  (April  -  July  2007) 


■  Evaluated  several  alternatives,  and  selected  JSAF  as  simulation  of 
choice  to  model  tactical  vignette  in  selected  “100  x  100  mi.  box” 

■  Obtained  /  installed  JSAF  simulation 

■  Modified  JSAF  sensor  module  to  model  radiation  detector 

■  Obtained  terrain  database  for  selected  area 

■  Set  up  “land  bridge”  scenario  vignette  and  checkpoint  in  JSAF 

■  Performed  multiple  JSAF  executions  to  generate  sensor 
performance  estimates,  including  multi-sensor  detections 

■  Tabulated  sensor  performance  data  in  spreadsheet  for  use  during 
table-top  exercise 


dPL 


Spiral  1  Activities  (August  2007  -  January  2008) 


■  Conducted  trade-off  study  to 
determine  most  effective 
configurations  of  dual  DTRA  and 
JHU/APL  systems  engineering  M&S 
laboratories 

■  Procured  hardware  and  software  for 
both  M&S  laboratories 


■  Instantiated  three  land-based  scenario  vignettes  for  checkpoints 
in  JSAF,  consistent  with  accepted  campaign-level  scenario 

■  Developed  higher  fidelity  radiation  detector  module  for  JSAF 

■  Performed  multiple  short  JSAF  executions  to  get  (preliminary) 
performance  curves  for  a  variety  of  passive  radiation  sensor 
types 


Spiral  2  Activities  (February  -  September  2008) 


■  Instantiated  five  additional  scenario  vignettes  in  JSAF 

■  Incorporated  3D  rendering  of  vignettes  using  JStealth,  federated  with 
JSAF 

■  Improved  JSAF  passive  radiation  detection  module 

■  Developed  a  JSAF  module  to  model  active  concepts  for  nuclear  material 
detection 

■  Performed  additional  JSAF  runs  to  explore  the  performance  of  selected 
combinations  of  sensors  and  to  produce  inputs  for  Joint  Analysis  System 
(JAS)  campaign-level  simulation  executions 

■  Incorporated  intelligent  behavior  for  red  and  blue  assets  in  JSAF 

■  Federated  new  “Behavior  Module”  with  JSAF,  based  on  prior  JHU/APL 
“commander  federate”  Independent  Research  and  Development  (IRAD) 
project 

■  Incorporated  tactics  based  on  discussions  with  subject  matter  experts 

B  Developed  a  secure  shared  repository  containing  scenario  information, 
sensor  characteristics,  and  performance  results  from  simulation 
executions 


dPL 
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Constructive  Simulation  Environment 
Progress  Toward  Long-Term  Vision 


io  Vignettes  (Three  Types) 


■  Land-based  checkpoints  to  detect  mobile  nuclear  material 

■  Rural  /  mountainous,  limited  road  system 

■  Rural  /  flatland,  broader  road  system 

■  Port  area 

■  Land-based  detection  of  stationary  /  hidden  nuclear  material 

■  Rural  hideout 

■  Above-ground  storage  site 

■  Underground  facility 

■  Detection  of  mobile  nuclear  material  in  maritime  environment 

■  Straits 

■  Open  water 


dPL 
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Nuclear  Radiation  Detection  Modeling 
Source  Signal  to  Detector 


S(e,r) 


S  e~Air)r 

gross 


4  7Tt 


,2 


s(e,r) 


Radiation  Spectra  Present  at 
Detector  Face 

s(e,r)  Fractional  spectral  density  present  at  the 
detector  face  (function  of  energy  &  range) 
S(e,r)  Spectrum  present  at  detector  face 


Sensor  Model  Input  Constants 


Sensor  Model  -  Input  Constants 

B gross =  Gross  background  count  (counts/sec)  -  Assume  constant 
a  -  atmospheric  attenuation  coefficient. 

Properties  dependent  on  detector  type: 

^ sensor =  projected  surface  area  of  detector  [m2] 

FOV  =  field  of  view  of  sensor  [deg] 

sB  =  Gross  background  count  efficiency  [unitless] 

pE  =  Fraction  of  gross  background  count  in  energy  band  E  [unitless] 

ss  =  Gross  source  count  efficiency  [unitless] 

(jE  =  Fraction  of  gross  source  count  in  energy  band  E  [unitless] 


Sensor  Model  Functions 


Sensor  Model  -  Functions 


Calculate  background  signal,  <NB  E>  (assume  constant  Bgross) 


Calculate  source  signal,  <NS  E> 


Calculate  signal-to-noise  ratio  (SNR) 


Calculate  detection  probabilities,  PD,  PFA 


Random  draw  for  Detection  Outcome 


JSAF  Source-Sensor  Model  Data  Interactions 


Where: 

S gross  =  Gross  source  count 
(counts/sec) 

x  =  integration  time  interval  (sec) 

V  =  Relative  velocity  of  target  (m/s) 
r  =  Range  of  target  (m) 

and  “Detection  Outcome”  can  be 
one  of: 

-No  detection 
-Positive  detection 
-False  positive  detection 
-Negative  detection 
-False  negative  detection 


Sensor  Signal-to-Noise  Ratio  vs.  PD,  PFA 
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Nuclear  Radiation  Detection  Modeling 
Physics  for  Active  Detection 


GeV  Proton  Beam 


Attenuator  Structure  jarget 


Detector 


Materials 

Attenuators 

Structures 

Targets 

Shielding 

Air,  Graphite 

Iron,  Steel, 
Aluminum 

Uranium  (235,  238), 
Lead,  Tungsten, 
Carbon,  Calcium, 
Silicon 

Iron,  Lead,  Wood, 
Water 

Motivation  for  the  Behavior  Module 


Issue 


"  Standard  scripting  for  Red  and  Blue  CONOPS  in  scenario  vignettes  in 
JSAF  attributed  insufficient  reactive  behavior  to  humans  involved 

■  For  example,  drivers  of  vehicles  carrying  nuclear  materials  simply 
proceeded  to  known  checkpoints,  were  scanned,  and  detained 

Behavior  Module  needed  to 

■  Provide  reactive  CONOPS  for  Red  and  Blue  assets,  and  background 
behaviors  for  Green  entities 

■  Enable  analysis  of  effectiveness  of  Blue  CONOPS  using  various  sensors 
in  opposition  to  Red  CONOPS,  together  with  Green  background  activity 

■  Enable  trades  between  CONOPS  and  sensor  investment  decisions 

■  For  example,  given  a  sensor’s  maximum  probability  of  detection,  can 
an  adjustment  of  resources  to  execute  a  CONOPS  improve  the  ability 
to  detect  and  interdict  the  nuclear  material? 


dPL 
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Technology  Basis  for  Behavior  Module: 
Hybrid  Reasoning  Technology  Framework 


Hybrid  Reasoning  Framework 
Preparation  and  Use 

■  Analyst  gains  understanding  of  the  decision  process  to  be  simulated 

■  Analyst  maps  appropriate  reasoning  technologies  to  the  decision  process 

■  Analyst  creates  a  domain-specific  file  using  reasoning  software  GUI  editor  and 
exports  the  domain-specific  file  for  agent’s  future  use 

■  Agents  load  domain-specific  files  as  needed  for  their  function,  provide  problem 
specific  inputs  (state),  and  use  appropriate  generic  engines  to  create  solution 
outputs 
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Baseline  Concept  for  Enhanced  Checkpoint 
Scenario  Vignette  Using  Behavior  Module 


Behavior  Module 

Current  Blue  Checkpoint  Behaviors 


When  a  Red  or  green  vehicle  reaches  a  checkpoint,  a  roadblock  prevents 
passage  in  one  direction  until  the  sensor  reports 

■  If  the  reading  is  negative,  the  roadblock  raises  and  allows  the  vehicle  to 
pass 

■  If  the  reading  is  positive, 

■  The  vehicle  is  sent  to  a  quarantine  location 


-  A  mobile  sensor  platform  is  tasked  to  report  to  the  quarantine  location 
to  conduct  a  second  reading 

■  If  the  second  reading  is  negative,  the  vehicle  is  allowed  to  continue  its  journey 

■  If  the  second  reading  is  positive,  the  vehicle  is  sent  to  a  holding  location. 

"  Blue  adjusts  the  roadblock  to  manage  the  length  of  the  traffic  backup 
■  Certain  types  of  vehicles  are  randomly 
selected  for  checks.  If  the  checkpoint 
queue  gets  too  long,  Blue  communicates 
to  the  Blue  selection  point  in  order  to 
reduce  the  percentage  selected  until  the 
queue  length  is  satisfactory. 


ujnunra 

mmv 
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Behavior  Module 

Current  Red  Checkpoint  Behaviors 

■  Red  vehicle  starts  at  safe  house  and  travels  to  destination  over 
existing  roads 

■  Red  vehicle  is  informed  by  a  Red  “scout”  vehicle  that  a  checkpoint 
is  ahead  so  that  the  Red  vehicle  with  weapons  grade  nuclear 
material  either  aborts  or  evades 

■  When  a  Blue  checkpoint  is  seen,  Red  has  option  to: 

■  Stop  and  pull  over  at  the  roadside  for  ten  minutes  to  consider 
either  aborting  or  continuing 

■  Continue  journey 

■  Evade  around  checkpoint  if  possible 

■  Abort  and  return  to  starting  location 

■  Red  either  diverts  or  aborts  if  progress 
along  the  current  route  is  too  slow. 


Behavior  Module 

Current  Green  Checkpoint  Behaviors 

■  Background  traffic  without  any  nuclear  sources 

■  Several  green  vehicles  with  medical  nuclear  sources  are 
dispatched  at  random  intervals  to  add  to  congestion  at 
checkpoints 


Near-Term  Plans 

Spiral  3  (October  2008  -  September  2009) 

"  Enhance  passive  nuclear  detection  modeling 

■  Spectroscopic  capability  for  source  and  sensor 

■  Gamma  imaging  capability  for  sensor 

■  Passive  fast  and  thermal  neutron  imaging  for  source  and  sensor 

■  Thermal  neutron  directional  /  imaging  capability  for  sensor 

■  Fast  neutron  imaging  for  sensor 

■  Add  fidelity  to  active  nuclear  detection  concept  modeling 

■  Introduce  active  interrogation  sources 

■  Modify  targeted  material  behavior 

-  Modify  passive  sensor  as  needed  to  support  active  concepts 

"  Add  behaviors  to  Behavior  Module  (selected  based  on  perception  of  Red 
capabilities) 

■  Blue  -  downstream  checkpoints,  traffic  funneling  to  checkpoints,  vehicle 
tagging  and  tracking,  CONOPS  variation  based  on  nature  of  Blue  forces 

■  Red  -  Red  security  vehicle  follower,  bribery,  rush-hour  exploitation, 
peaceful  demonstration,  traffic  accident  diversion,  limited  attacks,  etc. 


Systems  Engineering  Plan  and 
Systems  Engineering  Management 

Plan  Alignment 


NDIA  11th  Annual 
Systems  Engineering  Conference 

October  21, 2008 


Chet  Bracuto 

DoD  OUSD  A&T  (SSE) 


Bob  Scheurer  P.E.,  P.M.P. 

Boeing  Integrated  Defense  Systems 


Purpose 


Present  efforts  of  SE  Working  Group  discussions 
with  recommendations  for  improving  Acquirer  and 
Supplier  technical  planning 


Outline 


■  Problem  Definition 

■  Background 

■  Future  State 

■  Approach 

■  Traits  of  SEPs  and  SEMPs 

■  SEP  -  SEMP  Comparisons  and  Findings 

■  Vision  of  the  Ideal  SEMP 

■  Data  Item  Description  Update 

■  Benefits 

■  Way  Forward 

■  Questions/Answers 


Problem  Definition 


The  Need: 

■  Improved  SE  planning  discipline  to  better  facilitate  program 
execution 

■  Better  communication,  integration,  and  efficiency  between  acquirer 
and  suppliers 

■  Early  technical  planning  (i.e.  in  RFP)  to  ensure  that  SE  is  scoped 
and  priced  adequately  in  the  contractors’  proposals 

■  Better  planning  alignment  between  acquirer  and  suppliers 


Programs  Need  Improved  Guidance 
That  Will  Yield  More  Effective  Planning 


Background 


■  Systems  Engineering  Plan  (SEP)  is  a  DoD-developed  (acquirer) 
technical  planning  document  required  for  milestone  approval 

■  Systems  Engineering  Management  Plan  (SEMP)  is  a  contractor- 
developed  (supplier)  plan  for  the  conduct,  management,  and  control 
of  the  integrated  engineering  effort 

■  DoD  SEP  Preparation  Guide  was  updated  in  October  2007  to 
improve  completeness  and  consistency  in  SE  planning 

•  Highlighted  five  (5)  key  areas  of  SE  planning 

■  Briefed  NDIA  SE  Conference  in  October  2007  on  feasibility  of  a 
single,  unified  plan 

■  Questions  raised  if  other  DoD  policy  and  guidance  needed  updating 
(e.g.,  DI-MGMT-81024) 


Background 


Path  to  a  Unified  SE  Plan 
October  2007 


Unified 


SE  Plan 


Bidders’ 

Conference 


Aligned 
SE  Plans 


Supplier-Specific 
Lower-  Level 
Planning 


Proposal 

Submittal 


Contract 

Award 


I  i 

■  Update 

Update  Event  2 

Event  1  SRR 

IBR 


Update 
Event  n 


MSG07-1 161 94-009ppt 


Approach 


Evaluated  Feasibility  of  a  Unified  SE  Plan 

•  Launched  Study  to  Explore  the  Current  Environment 
on  Programs  Regarding  SEPs  and  SEMPs 

•  Selected  Five  Boeing  Programs  for  Review 

•  Gained  Understanding  of  Differences  and  Similarities 
Between  the  Two  Documents  (SEP  /  SEMP)  in  the 
Current  Environment 


Traits  of  the  SEP 


Defines  government  (customer)  technical  planning  expectations 

•  What  needs  to  happen  from  customer  perspective 

Describes  overall  approach  in  key  areas 

•  Requirements 

•  Technical  Staffing  and  Organizational  Planning 

•  Technical  Baseline  Management 

•  Technical  Review  Planning 

•  Integration  with  Program  Management 

Provides  contractor  guidance  for  systems  engineering  as  applied  to 
the  acquisition  program  at  hand 

Identifies  to  program  management  and  contract  personnel  the 
essential  systems  engineering  activities  and  products  required 


Traits  of  the  SEMP 


Responsive  to  the  contract  and  the  SEP 

Defines  contractor  (supplier)  technical  planning 

•  How  it  will  be  accomplished  from  the  contractor  perspective 

Contractor  further  develops  planning  outlined  in  the  SEP 

Project  (Supplier)  team  articulates  details  of  their 

•  Processes 

•  Tools 

•  Organization 

•  etc. 


Describes  activities  involved  in  the  transformation  from  requirements 
to  solution 


Includes  integration  of  subcontractor  planning 


SEP-SEMP 

Paragraph  Comparisons 


Common  Areas  of  Discussion 


Unique  Areas  of  Discussion 


A  Majority  of  SEP  Sections  Could 
Readily  be  Mapped  to  SEMP  Sections 


SEMP 


Specific  Findings  from 
SEP  &  SEMP  Comparisons 


■  SEP  and  SEMP  both  deal  with  SE  planning  but  from 
different  perspectives 

•  SEP  focus  is  acquirer  problem  space 

•  SEMP  focus  is  supplier  solution  space 

■  Documents  discuss  similar  subjects  but  are 
disconnected 

•  Different  language/terminology 

•  Different  paragraph  structures 


Alignment  of  Plans  is  Preferred  Over  Unification 


SEP-SEMP  Comparison 
Specific  Findings 


Over-all 

■  Stakeholders  are  different 

•  SEP:  Owner  is  Government  (Acquirer) 

•  SEMP:  Owner  is  Contractor  (Supplier) 

■  Details  are  different 

•  SEP:  Acquirer-focused  problem  definition 

•  SEMP:  Supplier-focused  solution  description 

■  Perspectives  are  different 

•  SEP:  Oversight  focus 

•  SEMP:  Delivery  focus 


SEP-SEMP  Comparison 
Specific  Findings 


Requirements 

•  Emphasis  is  different 

■  SEP:  Key  program  requirements 

■  SEMP:  Translating  requirements  into  product  deliverables 

Technical  Staffing  and  Organizational  Planning 

•  Differing  types  of  talent  needed  by  each  organization 

•  Organizational  interfaces  are  key  for  alignment 

•  Combined  organizational  details  are  unnecessary 

Technical  Baseline  Management 

•  Different  focus 

■  SEP:  What  the  Baselines  are  (descriptions) 

■  SEMP:  Achievement  of  the  Baselines  with  Supporting 
Processes 


SEP-SEMP  Comparison 
Specific  Findings 


■  Technical  Review  Planning 

•  Common  interests 

•  Different  preparation  approach 

■  SEP:  Review  Strategy;  What’s  to  be  Reviewed 

■  SEMP:  ‘How’  it’s  Reviewed;  ‘What’  is  deferred  to  the  IMP; 
‘When’  is  deferred  to  IMS 


■  Integration  with  Program  Management 
•  Different  detail  levels  and  focus 

■  SEP:  Integration  of  Planning  between  Government  and 
Contractor 

■  SEMP:  Total  Integration  of  Engineering  Effort  with 
Government  and  between  Contractor,  Associated 
Contractors,  and  Sub-Contractors 


Vision  of  the  Ideal  SEMP 


Used  regularly  by  the  program  for: 

•  Consistency  with  DoD  SEP 

•  Communicating  with  the  program  personnel 

■  How  things  get  done  on  the  program 

•  Maintaining  the  baseline  of  program  technical  planning  concepts 

•  Introducing  new  team  members  to  program  objectives 

Improves  program  efficiency  by: 

•  Creating  a  uniform  understanding  of  the  program  approach 

•  Establishing  a  common  program  lexicon 

•  Maintaining  support  of  the  technical  margin  (boundaries) 

Has  on-going  relevance  via 

•  Periodic  updates,  e.g.,  program  reviews 

•  Consistency  with  the  contractor’s  goals  and  environment 


SEP  Content  or  Paragraph  Leads  to 
SEMP  Content  /  Paragraphs  Containing  Supporting  Details 


Data  Item  Description  Updated 


■  DID  DI-MGMT-81024  (Systems  Engineering 
Management  Plan) 

■  Last  released  in  August  1990 

■  Based  on  MIL-STD-499A 

■  DID  outdated  due  to  changes  in  DoD  acquisition 
environment,  lessons  learned,  references,  etc. 

■  DID  drives  contractor  to  divert  from  newer  Government 
SE  policy  and  guidance 


Data  Item  Description  Updated 


■  Team  assembled  June  2008  to  investigate  possible 
improvements 

•  Emphasis  to  align  SEMP  DID  with  the  SEP  Prep 
Guide  Topics 

•  Team  consisted  of  OSD  and  Services 


Data  Item  Description  Updated 


10  PREPARATION  INSTRUCTIONS 


Draft  DI-MGMT-81024 
Update 


1 )  Alignment  with  SEP  Prep  _ 

Guide  Topics 

2)  Alignment  with  Program  SEP 

3)  Contractor-Specific  Planning 

4)  Plan  Completeness  - 

5)  Planning  Flexibility  - 


10.1  Format.  The  SEMP  format  shall  be  selected  by  the  contractor.  Unless  effective 
presentation  would  be  degraded,  the  initially  used  format  arrangement  shall  be  used  for 
subsequent  submissions. 

10.2  Content.  The  SEMP  shall  describe  the  contractor's  planned  systems  engineering 
processes  and  approach,  tailored  as  necessary  to  the  program's  contract,  objectives,  and 
overall  technical  and  management  approach.  The  SEMP  shall  describe  the  contractor's 
detailed  operational  plan  for  executing  systems  engineering  and  include,  at  a  minimum 
the  following  SE  related  topics  and  processes: 

■  The  topics  detailed  in  the  latest  version  of  the  Systems  Engineering  Plan  (SEP) 
Preparation  Guide  as  put  forth  by  the  Office  of  the  Secretary  of  Defense,  Systems 
and  Software  Engineering  Directorate. 

■  Government  planning  as  detailed  in  the  government  SEP. 

— ►  ■  Planning  associated  with  application  of  the  contractor's  systems  engineering 
processes  as  tailored  to  the  program  and  at  a  level  of  detail  necessary  for  the 
contractor  to  manage  and  execute  the  technical  effort. 

—►  ■  Referenced  lower-level  and  subcontractor  technical  plans,  for  example  in  the 
areas  of  risk  management,  requirements  management,  or  configuration 
management,  as  determined  necessary  by  the  contractor  to  plan  and  execute  a 
total  systems  engineering  effort. 

— ►  ■  Other  areas  deemed  necessary  to  execute  systems  engineering  to  meet  the 

program's  contract,  objectives,  and  overall  technical  and  management  approach. 


11.  DISTRIBUTION  STATEMENT 

DISTRIBUTION  STATEMENT  A:  Approved  for  public  release;  distribution  is 
unlimited. 


New  DID  Update  Strengthens  Alignment  Between  SEP  and  SEMP 


Alignment  via  the  Update  of  . 
SEMP  DID  (DI-MGMT-81024) 


SE  Plan 
Prep  Guide 


Contractor’s 
SEMP  Guide 


i 


Customer 

Implementation 


Contractor 

Implementation 


SEPs 


i 


Aligning  future  SE  planning 
involves  adjusting  the  DID  focus 
with  the  SEP  Prep  Guide 


SEMPs 


1 43030-002. ppt 


Future  State 


Path  to  a  Seamlessly  Aligned  Set  of  SE  Plans 

October  2008 


Seamlessly  Aligned 


Acquirer 

Draft  SE  Planning 


Aligned 
SE  Plans 


SE  Plans 


Supplier 
Draft  SE  Planning 


RFP  Release 


Proposal 

Submittal 


Contract 

Award 


Update 
Event  1 

IBR 


■ 

Update 
Event  2 

SRR 


Supplier-Specific 
Lower-  Level 
Planning 


► 


Source  Transfer  of  Planning  Information 
Feedback  Transfer  of  Planning  Information 


Benefits  of 

SEP  -  SEMP  Alignment 


Two  good  stand  alone  documents  can  be  far  better  with 
alignment 

■  Consistent  planning 

■  Reduction  in  duplication 

■  Reasonable  standardization 


Continuity  across  plans 


Way  Forward 


■  Distribute  Draft  DI-MGMT-81024  for  Industry  and  Government 
Comments 

■  Consider  Piloting  on  Programs 

■  Revise  and  Release  DI-MGMT-81024 

■  Change  Contractor  Guidance  in  Response  to  Updated  DID 

■  Monitor  Implementation  and  Feedback  from  Programs 


Questions/Answers 


Does  this  approach  appear  viable? 

What  improvements  would  you  like  to  see? 

What  other  recommendations  would  you  make  to  achieve 
aligned  planning? 


Backup/Reference  Material 


SEP-SEMP  Summary 


SEP  Prep  Guide 

Program  SEP  Comments 

Program  SEMP 

1.  Introduction 

Consistent  with  SEP  Prep  Guide 

Consistent  with  program  SEP 

2.  Program  Requirements 

Consistent  with  SEP  Prep  Guide 

1.  SEMP  covers  SEP  requirements 

2.  SEMP  addresses  design  considerations  in  program 
plans  and  directives  (section  8).  Which  are  detailed  plans. 

3.  Technical  Staffing  and 
Organizational  Planning 

Consistent  with  SEP  Prep  Guide 

1.  SEP  and  SEMP  are  consistent 

4.  Technical  Maturation  and 
Planning 

SEP  Prep  Guide  emphasizes  Requirements 
management  and  traceability  while  Program  SEMP 
describes  the  SE  process  and  RA/RM  in  context  of  the 

SE  process. 

1 .  SEMP  has  a  strong  description  of  how  the  SE  process  is  adapted  to  the 
program. 

5.  Technical  Review  Planning 

Program  SEP  provides  good  detail  on  technical 
reviews. 

Doesn’t  appear  to  be  covered  in  detail.  References  MIL-STD-1521B,  May  be 
covered  in  a  detailed  plan  such  as  Guality  Assurance  Plan. 

6.  Integration  with  Overall 
Program  Management 

Program  SEP 

Mostly  not  covered  in  the  SEMP.  Does  provide  a  brief  mention  of  the  use  of  the 

IMP  and  IMS  and  application  to  Risk  Management.  This  potentially  deserves  a 
stronger  emphasis.  For  example  there  is  not  mention  of  the  WBS. 

Section  8  Plans  and  Directives  Process  and  Products  -  This  section  references 
more  detailed  plans. 

I  I  Represent  Gaps 


Specific  Findings 

Requirements 


SEP 


Output  is  management  requirements 

Over-all  architecture  for  program 
lifecycle 

Emphasizes  program  requirements 
specifics 

•  KPPs 

•  MOEs,  e.g.  Reliability  or 
Maintainability 

•  Spiral  Outs 

•  Capabilities 

•  Etc. 

Defines  lifecycle  readiness  of 
capabilities  /  requirement  maturities 


SEMP 

Executable  process  for  how  technical 
management  is  done  on  the  program 

Defines  the  process  to  develop  the 
requirements,  not  the  actual 
requirements 

Emphasis  on  SE  Process  for  Analysis 

Identification  of  participants  in 
requirements  process 

Methods  for  transforming  abstract  to 
real 

Built  around  WBS  structure 
Integration  of  all  subordinate  plans 


Specific  Findings 

Technical  Staffing  &  Organizational  Planning 


SEP 


Acquirer-centric 

•  Govt.  IPT  Structure 

•  OIPTs 

•  WIPTs 

•  Govt  LSI  IPTs 

Associated  High-Level  Contractor 
IPTs 


SEMP 

Supplier-centric 

•  Contractor  and  Supplier 
IPT  Structure 

Program  organizational  structure 

•  Subordinate  considerations 
to  program  plan 

Partnerships 


Critical  Skills 


Specific  Findings 

Technical  Baseline  Management 


SEP 


SEMP 


Configuration  Management  /  Data  Specific  configuration 

Management  Activities  changes/updates  to  system 


Responsible  Entities 


Interface  management 


Specification  Tree  Supplier-specific  change 

management  processes 

Use  of  Technical  Baseline  and 

Technology  Readiness  Change  review  boards 

Assessments 


Identification  of  Relevant  DIDs 


Specific  Findings 

Technical  Review  Planning 


SEP 


SEMP 


Event-driven  technical  reviews 

Management  of  reviews 

Chairs,  stakeholders 

Facilitation  of  participation 

Past  Accomplishments  and 
Future  Expectations 


Consistency  with  IMP 

Events  and  Associated  Reviews 
summary 

Review  Planning  may  rely  on 
content  of  superior  documents 
(e.g.,  Program  Execution  Plan) 


Specific  Findings 

Integration  with  Program  Management 


SEP  SEMP 


Integration  with  other  planning 

•  Acquisition  Strategy 

•  IMP/IMS 

•  External  Functions 

•  Use  of  Technical  Review 
Results  (e.g.,  Baselines) 

Execution  requirements  for  SE 
activities 

•  Risk  Management 

•  T&E  Integration 

•  Verification  &  Validation 
Plan  integration 

•  TEMP  Traceability  to 
Performance  Reqmts 


Integration  between  program 
stakeholders 

•  Suppliers 

•  IPTs 

•  Customer 

•  Associate  Contractors 

Integration  of  the  engineering 
effort 

More  detailed  planning 

•  Scheduling 

•  Process  integration 

•  Subcontract  management 

•  Risk  management 


SE  Planning  Alignment  Vision 

Maturation  Sequence 


Govt.  PM  Office  Contractor 


Continues  for  MS  C  and  Full  Rate  Production 


New  Concepts  and  Trends 

-  How  Future  Trends  in  Systems 
and  Software  Technology  Bode  Well 
for  Enabling  Improved  Acquisition  and 
Performance  in  Defense  Systems 


11th  Annual  Systems  Engineering  Conference 
October  20-23,2008 
Hyatt  Regency  Mission  Bay 
San  Diego,  CA 

Theme:  Technology  -  Tipping  the  Balance 


Dr.  Kenneth  E.  Nidiffer 
Director  of  Strategic  Plans  for 
Government  Programs 

nidiffer@sei.cmu.edu 

703.908.1117 


Software  Engineering  Institute 


Carnegie  Mellon 
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The  Software  Engineering  Institute  -  Improving  the  Practice 
of  Engineering:  Create,  Apply  and  Amplify 


Federally  Funded  Research  and  Development  Center 
Created  in  1984 


Sponsored  by  the  U.S.  Department  of  Defense 

Locations  in  Pittsburgh,  PA;  Washington,  DC; 
Frankfurt,  Germany 


Operated  by  Carnegie  Mellon  University 
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Overview 


•  Transformational  Trends 

-  Development 

-  Acquisition 

—  Human  Element 
—  Risk  Management 
—  Communications 

•  Ten  Future  Trends 

•  Wrap-up 


“Perfect  Storm”  Event,  October  1991 

National  Oceanic  &  Atmospheric  Administration 
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Development:  Need  for  Space,  Air,  Ground,  Water, 
Underwater  Software-Intensive  Systems  that  are 
Interconnected 


Several  million  SLOC  programs;  “Hybrid” 
systems  combining  legacy  re-use,  COTS, 
new  development 

Multi-contractor  teams  using  different 
processes;  dispersed  engineering, 
development  &  operational  locations 

New  technologies  create 
opportunities/challenges; 
products  change/evolve,  corporations  mutate 

Business/operational  needs  change  -  often 
faster  than  full  system  capability  can  be 
implemented 

Skillset  Shortfalls;  Cost  and  schedule 
constraints 

Demands  for  increased  integration, 
interoperability,  system  of  system  capabilities 

Enterprise  perspectives/requirements; 
sustainment  concerns 


Development  Complexity  of 
Software-Intensive  Systems 
is  Increasing 
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Technology  Capability 


The  Acceleration  of  Innovation  in  the  21st  Century: 
-  Impacting  Both  Defense  and  Society 
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Augustine's  Law:  Growth  of  Software  -  Order  of 
Magnitude  Every  10  Years 


In  The  Beginning 
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1000 

LOC 
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F-15A 

50,000 

LOC 


1980's 


F-16C 

300K 

LOC 


1990's 
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Trend  &  Implications:  Augustine’s  Law  Will  Hold 


2080?  F-50  -  4. 7B  Lines  of  Code 


Need  for  increased  functionality  will  be  a  forcing  function  to  bring  the 
fields  of  software  and  systems  engineering  closer  together 
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Software  Engineering  Institute 


Moore's  Law:  The  Number  of  Transistors  That  Can  be 
Placed  on  an  Integrated  Circuit  is  Doubling 
Approximately  Every  Two  Years 


Integrated  Circuit  Complexity 


Transistors 
Per  Die 
1010-, 


106 

105 

104 


♦  1 965  Actual  Data 

■  MOS  Arrays  ▲  MOS  Logic  1975  Actual  Data 
1975  Projection 

Memory  ^  16A 

A  Microprocessor  1M 

256K 


256M 

128M 


51 2M 


>  Itanium™ 

y, _ *  Pentium®  4 

Pentium®  III 


A-  Pentium®  II 

Pentium® 
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4004 


8086 


30  1985  1990  1995  2000  2005  2 


Source:  Intel 
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Increased  Technological  Rate  of  Adoption 


Electricity 


■a 


Automobile  =  56 
years 


Television 

(1926) 


Telephone 


Telephone  =  36  years 
Television  =  26  years 

Cell  phone  =  14  years 

PC 
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Acquisition:  Life  of  a  Program  Manager  in  a  System  of 
Systems  and/or  Net-Centric  Operation... 
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A  Key  Challenge  is  How  to  Obtain  a  Better  Alignment  of  Risk  Among 

the  Relevant  Stakeholders 
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Acquisition  Performance  -  Flexible 
Boundary-Crossing  Acquisition  Structure 


2005  study  confirmed*: 

•  In  advanced  knowledge-based  organizations,  management’s  desire  for 
the  flow  of  knowledge  is  greater  than  the  desire  to  control  boundaries 

•  Unlike  the  matrix  organization,  there  is  less  impact  on  the  dynamics  of 
formal  power  and  control 

•  Important  to  measure  the  system  in  terms  of  user  performance 

*  Using  Communities  of  Practice  to  Drive  Organizational  Performance  and  Innovation,  2005,  APQ  study 


“Acquisition” 


Advanced  Knowledge-Based  Organizations  (Big  A) 


▼ 


“acquisition” 


From  “Science  and  Technology  to  Support  FORCE 

by  permission. 


Ref:  Jim  Smith,  (703)  908-8221  .jds@sei.cmu.edu 


s=r  Software  Engineering  Institute  CarnegieMellon 


How  Future  Trends  in  Systems  and  Software 
Engineering  Technologies  Bode  Well  for  Enabling  the 
Military  Mission  12 

Dr.  Kenneth  E.  Nidiffer 


©2008  Carnegie  Mellon  University 


Human  Element 


The  ability  of  organizations  to  compete  will  increasing  depend  on  the 

innovation  of  the  human  element 


fOMMAWI 

Society  Drivers:  Bimodal  Demographics  (Space  Industry)  ^ 


Reconstituting  This  Group 


Graduate 

School 

Shortfall 


Area  of  Concern 

A 


13% 


6% 


1% 


Area  of  Concern 

- A - 

11%, 


3% 


54%  of  the  S&  T  Workforce  is 
Over  45  and  wifi  be  retirement 
eligible  in  5  years 


2:1% 


14% 


10% 


3% 

IB-24  25-29  3Q-34  35-39  40-44  45-49  50-54  55-59 


Average  Space  Industry  S&E  Workforce  Age  Distribution 

Trend:  Industry/Gov’t  Will  Increasingly  Focus  on  Attracting,  Training 

and  Retaining  Systems  Engineering  Talent 


Software 


How  Future  Trends  in  Systems  and  Software 
Engineering  Technologies  Bode  Well  for  Enabling  the 


■  Source:  Lockheed  Martin  (0004305-001 :  AIAA  SE  Workforce  Data.  Frank  Cappuccio  VP  &  GM  Skunk  Works) 


Objective  is  for  Software  and  Systems  Engineering 
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Systems  Engr. 


System 

Design 


Systems 


Engineering  (SE) 


System 

Integrated 

Testing 


Systems  Engr. 


SW  Systems  Engr. 


Software  (SW) 
Requirements 
Analysis 


SW  Systems 
Engineering 


SW  System 
Testing 


Architectural 
SW  Design 


SW  Integration 
Testing 


SW  Systems  Engr. 


SW  Engineering 


Detailed  SW 

SW  Subsystem 

Design 

Testing 

SW  Engineering 


Code  and 

Unit  Test 

OSD  Initiative:  Integrated  Software  and  Systems  Engineering  Curriculum 
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Human  Element  in  the  Work-Space  Environment 


U1 


Timeline 


baby  boomers 

i**4<y  » ven 


millennial*! 

X  <V80I'WJ 


veterans 

»«2  V 
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Source:  Doug  Phair;  Technology  Evangelist; 

dohair@mitre.org:  February  2008 
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Communication:  Increased  Capabilities  in  the  Digital 
Spectrum  Enables  Improvements  in  Communication  and 
Collaboration 


Kilobit/Sec.  Megabit/Sec.  Gigabit/Sec.  Terabit/Sec. 

1,000  -103  1,000,000 -106  1,000,000,000- 109  1,000,000,000,000 -1012 


Rule  #4:  The  best  companies  are  the  best  collaborators * 

*  Friedman,  Thomas  L.  “  The  World  Is  Flat”,  Farrar,  Straus  and  Giroux,  2005 


s=r  Software  Engineering  Institute  CarnegieMellon 


How  Future  Trends  in  Systems  and  Software 
Engineering  Technologies  Bode  Well  for  Enabling  the 
Military  Mission  17 

Dr.  Kenneth  E.  Nidiffer 


©2008  Carnegie  Mellon  University 


Higher-Maturity  Approaches  to  Process  Improvement 
Are  Important  and  Synergistic  Trends 


Data-Driven  (e.g.,  Six  Sigma,  Lean) 


Model-Driven  (e.g.,  CMM,  CM  Ml) 


Optimizing 


Determine  what  your  processes  can  do 
(Voice  of  Process) 


•  Statistical  Process  Control 

Clarify  what  your  customer  wants  (Voice 
of  Customer) 

•  Critical  to  Quality  (CTQs) 

Identify  and  prioritize  improvement 
opportunities 

•  Causal  analysis  of  data 

Determine  where  your 
customers/competitors  are  going  (Voice 
of  Business) 

•  Design  for  Six  Sigma 


Quantitatively  Managed  ‘ 

Determine  the  industry  best  practice 

•  Benchmarking,  models 

Compare  your  current  practices  to  the 
model 


•  Appraisal,  education 

Identify  and  prioritize  improvement 
opportunities 

•  Implementation 

•  Institutionalization 

Look  for  ways  to  optimize  the  processes 


CMMI  and  Six  Sigma, 

Siviy,  et  al,  2007,  Addison  Wesley 
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Systems  and  Software  Engineering:  Ten  Trends 

1.  Greater  demands  on  systems  and  software  engineers  will  stimulate 
growth  in  the  field  -  nationally  and  internationally 

2.  Industry/Gov’t  will  increasingly  focus  on  attracting,  training  and  retaining 
systems  and  software  engineering  talent  -  short  and  long  run  -  with 
emphasis  on  providing  a  Generation  Y  work  environment 

3.  Increased  reliance  on  systems  and  software  engineering  processes  and 
technologies  to  effectively  manage  the  acquisition/”green”  space 

4.  The  laws  of  Augustine’s  and  Moore  will  continue  to  hold  and  will  continue 
to  be  a  forcing  function  to  bring  the  fields  of  software  and  systems 
engineering  closer  together 

5.  Improvements  risk-reduction  collaboration  mechanisms  will  be  significant 
enablers  for  increases  in  systems  and  software  engineering 
communication  and  “decision  velocity ” 
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Systems  and  Software  Engineering:  Ten  Trends 

6.  Systems  and  software  engineers  will  continually  find  way  to  innovative  to 
reduce  complexity 

7.  Increased  importance  of  modeling  and  simulation 

8.  Increased  customer  requests  for  system  and  software  engineering  support  will 
occur  earlier  in  life  cycle 

9.  Shift  of  systems  and  software  engineering  focus  from  the  platform  to  the 
networks  and  ground  systems 

10.  Process  improvement  will  continue  to  be  important! 
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Acquisition,  Technology  and  Logistics 


ESOH  I  n  Acquisition 

OSD  Expectations  For 
Implementing  DoDI  5000.02 

National  Defense  Industrial  Association 
1 1th  Annual  Systems  Engineering  Conference 
San  Diego,  California 
October  21,  2008 

Ms.  Karen  Gill  (Booz  Allen  Hamilton)  for 
Ms.  Patricia  Huheey 

Acquisition  ESOH 

Office  of  the  Deputy  Under  Secretary  of  Defense 
(Installations  &  Environment) 


Outline 


Acquisition,  Technology  and  Logistics 


•  Background  on  DUSD(I&E) 

•  Policy  Objectives  &  Principles 

•  Policy  &  OSD  Expectations 

•  Initiatives  and  Focus  Areas 

-  DAU  CLE  009  -  “System  Safety  in  Systems 
Engineering” 

-  “System  Safety  -  ESOH  Management  Evaluation 
Criteria  for  DoD  Acquisition” 

-  “ESOH  in  Acquisition  -  Integrating  ESOH  into 
Systems  Engineering”  Booklet 
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Background  on  ODUSD(l&E) 


ESOH  in  Acquisition  Leadership 

Environment,  Safety,  and  Occupational  Health 

Acquisition,  Technology  and  Logistics 


Background  on  ODUSD(l&E) 


Role  in  Acquisition 

Acquisition,  Technology  and  Logistics 

•  DUSD(I&E)  is  the  AT&L  Advisor  for  ESOH  issues 

•  Oversight  of  AC  AT  ID,  IAM,  and  AT&L  Special 
Interest  programs 

•  Focus  on  DoDI  5000.02  -  ESOH  in  acquisition  policy 

•  Identify  OSD  ESOH  “expectations”  in  the  Defense 
Acquisition  Guidebook 

•  Provide  guidance  for  policy  implementation  on  the 
Acquisition  Community  Connection 

•  Provide  ESOH  input  to  CJCS  3170.01  series  -  JCIDS 


Policy  Objectives  &  Principles 

Why  Be  Concerned  With 
ESOH  in  Acquisition? 

Acquisition,  Technology  and  Logistics 

•  ESOH  considerations  affect  the  operational 
effectiveness  and  sustainability  of  the  system 

-  There  is  a  relationship  between  the  natural  and 
workforce  infrastructures  and  the  military  mission 

-  Compliance  requirements  and  encroachment  influence 
how  DoD  maintains  and  trains  with  the  system 

-  System  design,  operation,  and  maintenance  parameters 
determine  the  installation  and  workforce  needs  to  train 
and  maintain  the  system 


Policy  Objectives  &  Principles 

Natural  Infrastructure 
Management  (Nl  M) 

Acquisition,  Technology  and  Logistics 


•  The  essential  principle  of  NIM  is 
that  air,  land,  and  water  resources 
are  assets  that  must  be  managed 
and  sustained  for  the  values  they 
provide  to  the  military 

•  Natural  infrastructure  (NI)  assets 
include 

-  Natural  assets:  distinct 
ecological  or  physical 
components  of  natural 
infrastructure 

-  Statutory  assets:  legally 
defined  entitlements  to  access 

and  use  products  and  services  Leveraging  Nl  asset  values  to 

of  natural  infrastructure  support  the  mission 


Policy  Objectives  &  Principles 

Plan  Ahead  &  I  nfluence 
the  System  Design 

Acquisition,  Technology  and  Logistics 

•  Identify  system  life-cycle  ESOH  risks  early  to  influence 
design  rather  than  address  them  afterwards  as  operational 
considerations 

•  System  design  is  most  effectively  influenced  through  the 
system  engineering  process 

-  Active  participation  in  the  IPTs  is  critical  to  success 

•  ESOH  hazards  and  associated  risks  are  best  managed 
using  a  standard  approach  and  structured  process 

•  E,  S,  and  OH  inputs  to  systems  engineering  need  to  be 
optimized  across  the  disciplines  to  meet  cost  and 
performance  needs  over  the  life  cycle 
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Policy  Objectives  &  Principles 


Top  Level  Principles 

Acquisition,  Technology  and  Logistics 

•  Address  safety  throughout  the  acquisition  process 

•  Use  a  total  systems  approach  to  minimize  or  eliminate 
characteristics  that  produce  environmental,  safety  or 
health  hazards 

•  Use  the  system  safety  process  in  MIL-STD-882D  to 
eliminate  ESOH  hazards  where  possible  and  manage 
ESOH  risks  where  hazards  cannot  be  eliminated 

•  Coordinate  ESOH  risks  with  the  User  and  formally  accept 
risks  at  designated  management  level 

•  Manage  and  document  hazardous  and  toxic  materials 
associated  with  the  system  and  plan  for  safe  disposal  8 


Policy  Objectives  &  Principles 


Life  Cycle  ESOH  Risks 

Acquisition,  Technology  and  Logistics 

•  ESOH  risks  may  include: 

-  Hazardous  and  toxic  materials  and  wastes 

-  Environmental  and  occupational  noise  (e.g.  litigation,  lost 
productivity) 

-  Personnel  safety  and  occupational  health  (e.g.  PPE,  medical 
surveillance,  lost  work  time,  future  VA  benefits  of  injury/illness) 

-  Regulatory  compliance  (e.g.,  pollution,  record-keeping,  non- 
compliance  fines,  litigation) 

-  System  component  or  software  failures 

•  Need  to  manage  ESOH  risks  associated  with: 

-  Routine  operation  and  maintenance  of  the  system 

-  System  failures 

-  ESOH  compliance  requirements 


Policy  &  OSD  Expectations 


DoD  Acquisition 
Policies  and  Guidance 

Acquisition,  Technology  and  Logistics 

■  DoD  Directive  (DoDD)  5000.01,  The  Defense 
Acquisition  System  (12  May  2003) 

■  DoD  Instruction  (DoDI)  5000.02,  Operation  of  the 
Defense  Acquisition  System  (12  May  2003) 

■  MIL-STD-882D,  DoD  Standard  Practice  for  System 
Safety 

■  Defense  Acquisition  Guidebook, 
https://akss.dau.mil/dag 

■  Acquisition  Community  Connection,  ESOH  Special 
Interest  Area,  https://acc.dau.mil/ESOH 
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Policy  &  OSD  Expectations 


Update  Policy  and  Guidance 

Acquisition,  Technology  and  Logistics 

•  In  2007  coordinated  with  the  Services  and  provided 
ESOH  input  update  of  DoDI  5000.02 

-  “Facts  of  life”  changes  only  (plus  AT&L  inputs) 

-  Incorporated  the  3  USD(AT&L)  ESOH-System  Safety 
Memos  since  2003  and  new  EO  1 3423 

-  Moved  main  ESOH  section  to  the  new  Enclosure  12  - 
Systems  Engineering  and  updated  ESOH  paragraphs 

•  Provided  updated  ESOH  section  to  the  Defense 
Acquisition  Guidebook  to  reflect  the  upcoming  changes 
to  DoDI  5000.02  (will  come  out  with  updated  5000.02) 
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Policy  &  OSD  Expectations 


Paragraph  3. 8. 2. 2 

Life- Cycle  Sustainment 
Considerations 

Acquisition,  Technology  and  Logistics  ■ 

•  Life-cycle  sustainment  considerations  include 
supply;  maintenance;  transportation;  sustaining 
engineering;  data  management;  configuration 
management;  manpower,  personnel,  training, 
habitability,  survivability,  environment,  safety 
(including  explosives  safety),  and  occupational 
health;  protection  of  critical  program  information 
and  anti-tamper  provisions;  supportability;  and 
interoperability. 
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Policy  &  OSD  Expectations 


Paragraph  3.8.3 

Disposal 

Acquisition,  Technology  and  Logistics 

•  At  the  end  of  its  useful  life,  a  system  shall  be 
demilitarized  and  disposed  of  in  accordance  with  all  legal 
and  regulatory  requirements  and  policy  relating  to 
safety  (including  explosives  safety),  security,  and  the 
environment. 

•  During  the  design  process,  PMs  shall  document 
hazardous  materials  contained  in  the  system  in  the 
Programmatic  Environment,  Safety,  and 
Occupational  Health  Evaluation  (PESHE)  (see 
paragraph  E12.6.),  and  shall  estimate  and  plan  for  the 
system’s  demilitarization  and  safe  disposal. 
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Policy  &  OSD  Expectations 

Enclosure  12  Systems  Engineering 

E12.6  Environment,  Safety,  and 
Occupational  Health 

Acquisition,  Technology  and  Logistics 


•  The  PM  shall  integrate  ESOH  risk  management 
into  the  overall  systems  engineering  process  for  all 
developmental  and  sustaining  engineering 
activities.  As  part  of  risk  reduction,  the  PM  shall 
eliminate  ESOH  hazards  where  possible,  and 
manage  ESOH  risks  where  hazards  cannot  be 
eliminated.  The  PM  shall  use  the  methodology  in 
MIL-STD-882D,  DoD  Standard  Practice  for  System 
Safety. 

•  PMs  shall  report  on  the  status  of  ESOH  risks  and 

acceptance  decisions  at  technical  reviews.  u 


Policy  &  OSD  Expectations 

Enclosure  12  Systems  Engineering 

E12.6  Environment,  Safety,  and 
Occupational  Health,  con't. 

Acquisition,  Technology  and  Logistics 

•  Acquisition  program  reviews  and  fielding  decisions 
shall  address  the  status  of  all  high  and  serious  risks, 
and  applicable  ESOH  technology  requirements. 

•  Prior  to  exposing  people,  equipment,  or  the  environment 
to  known  system-related  ESOH  hazards,  the  PM  shall 
document  that  the  associated  risks  have  been  accented 
by  the  following  acceptance  authorities:  the  CAE  for  high 
risks,  PEO-level  for  serious  risks,  and  the  PM  for  medium 
and  low  risks.  The  user  representative  shall  be  part  of 
this  process  throughout  the  life  cycle  and  shall  provide 
formal  concurrence  prior  to  all  serious  and  high  risk 
acceptance  decisions. 
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,  .  Policy  &  OSD  Expectations 

Enclosure  12  Systems  Engineering 

E12.6.1  Programmatic  ESOH 
Evaluation  (PESHE) 

Acquisition,  Technology  and  Logistics 

•  The  PM  for  all  programs,  regardless  of  ACAT  level,  shall  prepare  a 
PESHE  which  incorporates  the  MIL-STD-882D  process  and  includes 
the  following: 

-  identification  of  ESOH  responsibilities 

-  the  strategy  for  integrating  ESOH  considerations  into  the  systems 
engineering  process 

★  identification  of  ESOH  risks  and  their  status 

★  a  description  of  the  method  for  tracking  hazards  throughout  the 
life  cycle  of  the  system 

★  identification  of  hazardous  materials,  wastes,  and  pollutants 
(discharges/emissions/  noise)  associated  with  the  system  and 
plans  for  their  minimization  and/or  safe  disposal 

-  a  compliance  schedule  covering  qU  system  related  activities  for  the 
National  Environmental  Policy  Act  (NEPA)  and  Executive  Order 
12114 

•  The  Acquisition  Strategy  shall  incorporate  a  summary  of  the 

PESHE,  including  the  NEPA/EO  12114  compliance  schedule  16 


Policy  &  OSD  Expectations 

Enclosure  12  Systems  Engineering 

E12.6.2  National  Environmental  Policy 
Act/ Executive  Order  12114 

Acquisition,  Technology  and  Logistics 

•  The  PM  shall  conduct  and  document  NEPA/E.O. 

12114  analyses  for  which  the  PM  is  the  action 
proponent 

•  The  PM  shall  provide  system-specific  analyses  and 
data  to  support,  others  organizations  ’  NEPA  and  EO 

1211A.  analyses 

•  The  CAE  (or  for  joint  programs,  the  CAE  of  the  Lead 
Executive  Component)  or  designee,  is  the  approval 
authority  for  system-related  NEPA  and  E.O.  12114 
documentation 
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Policy  &  OSD  Expectations 


Enclosure  12  Systems  Engineering 

E12.6.3  Mishap  Investigation  Support 

Acquisition,  Technology  and  Logistics 

•  PMs  will  support  system-related  Class  A  and  B 
mishap  investigations  by  providing  analyses  of 
hazards  that  contributed  to  the  mishap  and 
recommendations  for  materiel  risk  mitigation 
measures,  especially  those  that  minimize  human 
errors. 
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Policy  &  OSD  Expectations 


Ml  L-STD-882D  Eight  Mandatory  Steps 


Acquisition,  Technology  and  Logistics 


1.  Document  the  system  safety  approach  - 

Document  the  Government  and 
Contractors’  approach  to  ESOH  risk 
management 

2.  Identify  hazards  -  Conduct  hazard 
analyses  of  ever-increasing  fidelity  as 
the  system  design  matures 

3.  Assess  the  risk  -  For  each  hazard, 
determine  the  associated  level  of  risk 

4.  Identify  risk  mitigation  measures  -  For 
each  identified  hazard,  propose 
altematives/controls  to  eliminate  the 
hazard  or  reduce  the  risk  of  the  hazard  to 
an  acceptable  level 


5.  Reduce  risk  to  an  acceptable  level  -  For 

each  hazard,  select  the  risk  mitigation 
measure(s)  to  be  used  to  eliminate  the 
hazard  or  reduce  the  risk 

6.  Verify  risk  reduction  -  For  each  hazard, 

verify  that  the  hazard  has  been  eliminated 
or  the  risk  mitigation  measure(s)  has 
reduced  the  risk  of  the  hazard 

7.  Review  hazards  and  accept  risk  by 

appropriate  authority 

8.  Track  hazards,  their  closures,  and  residual 

risk  -  Maintain  a  tracking  system  to 
document  hazards,  mitigation  measures, 
and  hazard  status  throughout  the  life  cycle 


OEp 


Policy  &  OSD  Expectations 


Ml  L- STD- 882 D  Order  of  Precedence 

Acquisition,  Technology  and  Logistics  ™ 


Most  to  Least  Preferred  Risk  Mitigation  Measures 

1. 

Eliminate  hazards  through 
design  selection 

If  unable  to  eliminate  an  identified  hazard,  reduce  the 
associated  risk  though  design  selection. 

2. 

Incorporate  safety  devices 

If  unable  to  eliminate  the  hazard  though  design  selection, 
reduce  the  risk  using  protective  safety  features  or  device. 

3. 

Provide  warning  devices 

If  safety  devices  do  not  adequately  lower  the  risk  of  the 
hazard,  include  a  detection  and  warning  system  to  alert 
personnel  to  the  particular  hazard. 

4. 

Develop  procedures  and 
training 

Where  it’s  impractical  to  eliminate  hazards  through  design 
selection  or  to  reduce  associated  risk  to  an  acceptable 
level  with  safety  and  warning  devices,  incorporate  special 
procedures  and  training.  Procedures  may  include  the  use 
of  personal  protective  equipment. 

Note:  For  catastrophic  or  critical  severity  categories,  avoid 
using  warning,  caution,  or  other  written  advisory  as  the 
only  risk  reduction  method. 

Initiatives  &  Focus  Areas 

“System  Safety  in  Systems 
Engineering"  DAU  CLE009 

Acquisition,  Technology  and  Logistics 

•  Roadmap  for  using  the  MIL-STD-882D  System  Safety 
methodology  to  integrate  ESOH  considerations  into  the 
SE  process  during  each  life  cycle  phase 

•  Maps  System  Safety  analyses  into  the  SE  “V”  model 

•  Never  been  done  before  by  either  the  System  Safety  or 
SE  communities — fundamental  breakthrough  in  defining 
how  the  communities  are  supposed  to  work  together 

•  Results  to  Date: 

-  Available  online  April  2005 

-  3054  graduates  as  of  OCT08 


ODUSD(l&E)  ■  ESOH  SME 
and  logistical  support 

ODUSD(A&T)/SSE  -  SE  SME 
and  DAU  SE  courses 


Initiatives  &  Focus  Areas 


System  Safety  -  ESOH  Management 
Evaluation  Criteria  for  DoD  Acquisition 

Acquisition,  Technology  and  Logistics 

•  Tool  to  assess  SE  technical  discipline  in  the  integration 
of  ESOH  using  System  Safety  methodology 

-  Technical  and  Program  Reviews  (self  assessment) 

-  Milestone  Review  Process  (oversight  assessment) 

•  Four  key  areas  for  evaluation 

-  Planning 

-  Requirements  Analysis 

-  Hazard  analysis 

-  Resources 
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Initiatives  &  Focus  Areas 


System  Safety  -  ESOH  Management 
Evaluation  Criteria  for  DoD  Acquisition 

Acquisition,  Technology  and  Logistics 

•  Assessment  criteria  for  each  area  for  each  life  cycle 
phase 

-  Weighted  summation  of  four  ratings  to  overall 
rating  for  each  life  cycle  phase 

•  Incorporated  into  the  next  Defense  Acquisition 
Program  Support  (DAPS)  SE  Assessment 
Methodology 

•  Available  at  Acquisition  Community  Connection 
https://acc.dau.mil/ESOH 
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Initiatives  &  Focus  Areas 


Integrating  ESOH  Into  SE  Booklet 


Acquisition,  Technology  and  Logistics 


Builds  on  CLE009 
and  depicts  when 
ESOH  activities 
should  be 
performed  to 
influence  system 
design  throughout 
the  systems 
engineering  process 

System  Safety- 
ESOH  Mgt. 
Evaluation  Criteria 
are  included  24 


Trish  Huheey 

ODUSD(l&E) 

ESOH  in  Acquisition  Lead 
(703)  604-1846 
patricia.huheey@osd.mil 


Acquisition,  Technology  and  Logistics 


BACKUP 


Nl  Asset  Valuation 


Acquisition,  Technology  and  Logistics 


•  Tenet:  To  make  good  decisions  about  managing 
assets,  it  helps  to  know  something  about  their  value! 

•  Corollary:  Value  depends  on  actual  or  potential  use. 
Therefore,  the  process  of  identifying  NI  asset  values 
can  uncover  innovative  opportunities  to  use  NI  assets 
to  support  the  mission. 

-  NI  asset  valuation  is  fundamentally  about  leveraging 
NI  asset  values  to  support  the  mission. 

•  NI  asset  valuation  is  a  set  of  approaches  and 
techniques  used  to  assess  values  of  NI. 
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Update  on 

Survey  on  Modeling  and  Simulation 

Support  for  the 

Systems  Engineering  of  Systems  of 

Systems 


Judith  Dahmann 
Jim  Hollenbach 


Systems  Engineering  for  Systems  of  Systems 


Systems  Engineering  Guide  for 
Systems  of  Systems 


Version  1.0 
August  2008| 


Director,  Systems  and  Stftware  Engineering 
Deputy  Uricfcr  Secretary  of  Defense  (Acquisition  and  Technology) 
Office  cf  the  Under  Secretary  of  Defense 
(Acquisition,  Technology  and  Logistics) 


SoS:  A  set  or  arrangement 
of  systems  that  results 
when  independent  and 
useful  systems  are 
integrated  into  a  larger 
system  that  delivers 
unique  capabilities 


AT&L  Released  “Systems  Engineering  for 
Systems  of  Systems”  Version  1 .0  in  August 
2008 

-  Characterizes  SoS  in  the  DoD  Today 

-  Identifies  core  elements  of  SoS  SE 

-  Discusses  application  of  SE  processes  to  SoS  SE 
core  elements 

-  Highlights  ‘emerging  principles’ 

-  Briefly  addresses  M&S 

Requested  M&S  Committee  provide  input 
on  use  of  M&S  to  support  SE  for  SoS 

Purpose  of  this  presentation  is  to 

-  Summarize  the  response  to  date 

-  Outline  plans  for  next  steps 

-  Solicit  additional  input  from  community 


Modeling  and  Simulation? 


How  does  the  SoS  SE  Guide  address  M&S? 

-  Initial  .9  Version  included  M&S  throughout  the  draft 

-  The  practitioner  reviews  indicate  limited  use  of  M&S 

•  Main  place  where  M&S  was  cited  is  in  the  emulation  of 
systems  not  otherwise  available  for  testing 

-  Consequently  the  1 .0  Working  Draft  limited  M&S  to 
this  area 

-  Comments  on  the  draft  identified  more  uses  of  M&S 

-  The  final  1.0  Version  has  an  M&S  section  and  added 
places  where  M&S  is  discussed 


Specific  Request 


For  each  of  the  seven  core  elements  of  SoS 
systems  engineering  (SE)  listed  on  the 
following  pages,  please  share  your  views  on: 

-  The  potential  for  applying  modeling  and  simulation, 
including  why  M&S  has  potential  value 

-  Your  experience  using  M&S  for  this  SoS  SE  element, 
including  the  context  of  the  application,  the  ways 
M&S  was  applied,  the  products  produced,  how  they 
were  used,  and  the  value  added  by  M&S 

-  The  enablers  for  use  of  M&S  in  this  element, 
including  what  attributes  made  successful  use  of 
M&S  possible  (in  cases  where  it  was  applied)  and 
barriers  that  inhibited  use  of  M&S  (in  cases  where 
the  potential  is  not  being  realized). 


Timeline 


•  Pre- Release  version  of  SoS  SEG  released  in  May  2008 

•  Dahmann  briefed  NDIA  M8tS  Committee  at  June  2008 
meeting 

•  June  letter  from  Kristen  Baldwin  (then  Acting  Director 
for  Systems  and  Software  Engineering)  requesting 
committee  input 

•  Survey  was  sent  out  by  Hollenbach  in  early  July 

•  Responses  were  received  in  September 

•  Next  steps  outlined  here 


How  did  we  get  here? 


Summary  of  Responses 


19  Responses  from  14  organizations 

10  volunteers  to  help  to  synthesize  the  report 
survey  results 

Responses  were  of  several  types 

-  Views  and  specific  experiences 

-  Perspective  on  issues 

-  Views  based  on  M&S  for  SE 

-  Organizational  experience 

8  responses  include  descriptions  of  specific 
activities  which  are  candidates  for  follow-up 


Responses  Overview 


Name 

Organization 

Indiv  or 
Org  input? 

Response  Type 

Exam¬ 

ple 

Help 

LTC  Favio  Lopez 

3CE 

Org 

Views  and  specific  experiences 

X 

X 

David  Dubuque 

Aegis  Technologies 

Indiv 

Perspective  on  issues 

Danny  Thomas 

Aegis  Technologies 

Indiv 

Perspective  on  issues 

Robert  Upchurch 

Aegis  Technologies 

Org 

Perspective  on  issues 

Hugh  Griffis 

Air  Force  Aeronautical  Sys  Ctr  (ASC/end) 

Org 

Organizational  experience 

X 

Terry  Christian 

Air  Force  Research  Lab  (afrl/xpt) 

Indiv 

Organizational  experience 

Pin  Chen 

Australia  DSTO  (Maritime  Ops) 

Indiv 

Views  based  on  M&S  for  SE 

William  Tucker 

Boeing  Company 

Org 

Views  and  specific  experiences 

X 

X 

Frank  Grange 

Lockheed  Martin 

Indiv 

Views  and  specific  experiences 

X 

X 

Steve  Hall 

Lockheed  Martin 

Indiv 

Views  and  specific  experiences 

X 

Chett  Harris 

Lockheed  Martin  (is&GS) 

Indiv 

Perspective  on  issues 

X 

Koury,  Polzer,  et.  al. 

Lockheed  Martin 

Indiv  but... 

Views  and  specific  experiences 

X 

X 

Lan-Than  McGough 

Marine  Corps  Systems  Command 

Org 

Views  and  specific  experiences 

X 

X 

Dave  Prochnow 

MITRE 

Indiv 

Views  and  specific  experiences 

X 

X 

George  Hazelrig 

National  Science  Foundation 

Indiv 

Views  based  on  M&S  for  SE 

- 

Steve  Lyda 

Naval  Air  Systems  Command  (Air  4.7) 

Indiv 

Views  based  on  M&S  for  SE 

Kenneth  Small 

Naval  Surface  Warfare  Cntr  Dahlgren 

Indiv 

Perspective  on  issues 

Thomas  Haley 

Naval  Undersea  Warfare  Center 

Indiv 

Views  based  on  M&S  for  SE 

X 

Emily  Andrew 

Raytheon 

Org 

Views  and  specific  experiences 

X 

X 

Government,  industry,  and  FFRDC  responses 


Proposed  Next  Steps 


Convene  team  of  volunteers  to  begin  synthesis 
processes 

-  Evening  meeting  during  SE  Conference  to  get 
started  (Wed  5:30-6:30  pm  in  this  room) 

Target  February  M&S  Committee  meeting  in 
DC  for  brief-out  of  draft  results 

Follow-up  with 

-  Written  report 

-  More  in-depth  look  at  specific  activities  which 
could  serve  as  examples 


Added  inputs  are  welcome 
Need  these  as  soon  as  possible 


Execution  of  the 
Acquisition  M&S  Master  Plan 

Progress  Report 


NDIA  Systems  Engineering  Conference 

San  Diego,  California 
20-23  October  2008 


James  W.  Hollenbach 
ODUSDfA  &  T)/SSE/D  TE  Support 

Simulation  Strategies,  Inc. 
jimh@simstrat.  com,  121. 824. 093 1 


Outline 


□  AMSMP  Development  (Review) 

□  AMSMP  Execution 

>  Funding  approach 

>  Progress  overview 

□  Future  Plans 

□  Q&A/Status  of  Individual  Actions 


Acquisition  M&S  Governance  Structure 


Industry 


SISO,  OMG,  etc. 
INCOSE 


SMAS,  SE  DSIG,  etc. 


NDIA 

Sys  Eng  Division 


INCOSE  MBSE  WG 


NDIA 

M&S  Committee 


DoD  Acquisition 


DoD  M&S 


Chair:  Ms  Philomena  Zimmerman 
PM  FCS  (BCT) 

Associate  Director, 

Modeling  and  Simulation 


Col  Sean  McAllum,  USAF 
Acquisition  Member: 
ODUSD(A&T)/SSE/DTE 


AMSWG  is  anchored  in  acquisition  community  and 
linked  to  industry  and  the  DoD  M&S  community 
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Acquisition  Modeling  &  Simulation  Master  Plan 


□  Purpose 

“Improve  M&S  support  to  the  DoD  acquisition  process...” 

□  Vision 

“Optimally  employ  responsive,  trustworthy,  and  cost-effective 
M&S  capabilities  to  support  defining,  developing,  testing, 
producing  and  sustaining  America 's  capabilities  that  support 
the  spectrum  of  DoD  missions.  ” 


Acquisition  M&S 


□  Definition:  M&S  used  to  help  define,  design,  develop,  test,  produce, 
operate,  and  sustain  defense  systems  and  systems-of-systems 

□  Scope:  Across  the  acquisition  life  cycle 


0 

Concept 

Decision 


Concept 

Refinement 


0 

MS' ‘A” 


Technology 

Development 


0 

MS“B” 

1 


System  Dev/ 
Demonstration 


0 

MS“C” 

1 


Production 


0 


O&S 


Full  Rate 
Prod  DR 

I 
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Potential  M&S  Benefits 


□  M&S  can  improve  design  (designs  are  models),  integration,  and  evaluation 

>  Accurately  track  complex  relationships  and  micro-level  interactions 

>  Present  macro-level  measures  of  merit  to  decision  makers 

>  Earlier,  more  accurate  understanding  of  a  system,  lowering  risk 

□  Means  to  deal  with  the  challenges  of  acquiring  capabilities/systems  of 
systems,  with  attendant  dramatic  increases  in  trade  space  and  complexity 

>  Track  the  many  more  entities,  variables,  interactions,  etc. 

>  Provide  a  shared  understanding  across  vast  development  enterprises 

□  M&S  can  speed  the  design-evaluation  cycle,  saving  time  and  money 

□  Provides  a  more  defendable  analytical  underpinning  for  decisions 

□  Credible  M&S  surrogates  for  systems  and  forces  can  cost-effectively... 

>  flesh-out  the  battlespace  for  live  tests  of  individual  systems 

>  provide  the  only  practical  way  to  assess  SoS  capabilities  as  they  evolve 
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AMSMP  Strategy 


□  Not  try  to  do  the  job  of  program/capability  managers;  rather, 
seek  to  empower  them  by 

>  Removing  systemic  obstacles  in  their  path 

>  Identifying  new  options  for  approaching  their  tasks 

□  Foster  widely-needed  M&S  capabilities  that  are  beyond  the  reach 
of  individual  programs 

□  Address  M&S  issues  and  actions  necessary  to  enable  acquisition 
of  joint  capabilities  (systems  of  systems) 

□  Lay  out  tasks  as  a  Work  Breakdown  Structure  (WBS) 

>  Discrete  tasks  with  identified  leads  and  explicit  deliverables 

>  Easier  to  resource,  schedule,  and  manage 

>  Each  contributes  to  better  M&S  support  to  acquisition 
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Acquisition  M&S  Master  Plan 
Development  Process 


Acquisition  M&S  Master  Plan 


Determine  &  Prioritize  What 
Acqn.  Community  Must  Do 


(Top-down) 


Desired  Acqn  Environment  per 
CJCSI  3170  &  DoDD  5000.1 


Identify  Needed  System 
Engineering  Capabilities 


Identify  Actions  of  Others 
(e.g.,  M&S  CO,  Nil,  NIST) 


Identify  Actions  Needed 
to  Address  the  Gaps 


Identify  M&S  Capability  Gaps 


Identify  Needed  M&S 
Capabilities 


Assess  Current  Issues/Needs 
(e.g.,  SoS  efforts) 


(Bottom-up) 


Assess  Recommendations  fm 
Prior  M&S  in  Acqn  Studies 


Assessment  Highlights 


□  Widespread  use  of  M&S  in  acquisition,  but  usually  stove-piped 

□  Many  M&S  representation  gaps  and  deficiencies 

□  Acquisition  staffs  mostly  uninformed  about  M&S  capabilities  and  limitations 

□  No  requirement  to  document  planned  M&S  support  to  acquisition 

□  No  effective  business  model  for  developing,  using,  and  maintaining 
broadly-needed  M&S  capabilities 

□  Weak  contractual  guidelines  for  M&S  and  data  needs 

□  Lack  of  agreed  standards  for  sharing  info  and  interoperating  M&S  tools 

□  Hard  to  discover  reusable  M&S  tools  and  data,  insufficient  info  to  evaluate 
reuse  candidates,  and  lack  of  reuse  incentives  =  little  reuse 

□  Virtual  ranges  (Live-Virtual-Constructive  simulation  environments)  aren’t 
readily  available 

□  VV&A  often  poor  or  non-existent;  weak  documentation  &  examination 
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Acquisition  M&S  Master  Plan  Structure 


*  Foreword 

*  Introduction 

•  Purpose 

•  Vision 

•  Scope 

*  Objectives  (5) 

*  Actions  (40) 

■  Action 

■  Rationale  (why  it’s  needed) 

■  Discussion  (implementation  guidance) 

•  Lead  &  supporting  organizations 

■  Products  (what  is  expected) 

•  Completion  goal  (year) 

*  Execution  Management 

http://www.acq.osd.mil/sse/as/quidance.html 
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Department  of  Defense 

Acquisition  Modeling  and 
Simulation  Master  Plan 


Issued  by  the 

DoD  Systems  Engineering  Forum 
April  17,  2006 


Five  Objectives,  40  Actions 


Objective  1 


Provide 
necessary 
policy  and 
guidance 


1-1  M&S 

management 

1-2  Model-based 
systems 
engineering  & 
collaborative 
environments 

1-3  M&S  in  testing 

1-4  M&S  planning 
documentation 

1-5  RFP  &  contract 
language 

1-6  Information 
Assurance 


Key 

Broader  than  Acqn 
Partially  broader 


Objective  2 


Enhance  the 
technical 
framework 
for  M&S 


2-1  Product 
development 
metamodel 

2-2  Commercial 
SE  standards 

2-3  Distributed 
simulation 
standards 

2-4  DoDAF  utility 

a)  DoDAF  2.0 
Systems 
Engineering 
Overlay 

b)  Standards  for 
depiction  & 
interchange 

2-5  Metadata 
template  for 
reusable 
resources 


Objective  3 


Improve 
model  and 
simulation 
capabilities 


3-1  Acquisition 
inputs  to  DoD 
M&S  priorities 

3-2  Best  practices 
for  model/sim 
development 

3-3  Distributed  LVC 
environments 

a)  Standards 

b)  Sim/lab/range 
compliance 

c)  Event  services 

3-4  Central  funding 
of  high-priority, 
broadly-needed 
models  &  sims 

a)  Prioritize  needs 

b)  Pilot  projects 

c)  Expansion  as 
warranted 


Objective  4 


Improve 
model  and 
simulation 
use 


4-1  Help  defining 
M&S  strategy 

4-2  M&S  planning 
&  employment 
best  practices 

4-3  Foster  reuse 

a)  Business  model 

b)  Responsibilities 

c)  Resource 
discovery 

4-4  Info  availability 

a)  Scenarios 

b)  Systems 

c)  Threats 

d)  Environment 

4-5  W&A 

a)  Documentation 

b)  Risk-based 

c)  Examination 
4-6  COTS  SE  tools 

4-7  M&S  in  acqn 
benefit  metrics 


Objective  5 


Shape  the 
workforce 


5-1  Definition  of 
required  M&S 
competencies 

5-2  Harvesting  of 
commercial 
M&S  lessons 

5-3  Assemble  Body 
of  Knowledge 
for  Acqn  M&S 

5-4  M&S  education 
&  training 

a)  DAU,  DAG  & 
on-line  CLMs 

b)  Conferences, 
workshops  & 
assist  visits 

5-5  MSIAC  utility 


Outline 


□  AMSMP  Development  (Review) 

□  AMSMP  Execution 

>  Funding  approach 

>  Progress  overview 

□  Future  Plans 

□  Q&A/Status  of  Individual  Actions 


Funding  Approach 


Prioritized  options  to  accomplish  AMSMP  actions 

1.  Accomplish  via  sweat  equity 

>  e.g.,  OUAD(A&T)/SSE  M&S  Cell  (resource  limited) 

2.  Compete  for  M&S  Steering  Committee  funds  (if  >  acqn) 

>  only  DoD-wide  M&S  Program  Element 

3.  Compete  for  OSD  funding  “targets  of  opportunity” 

>  e.g.,  study  funds,  end-of-year  sweep 

4.  Submit  as  SBIR  topics  (just  beginning) 

5.  Team  with  other  organizations 

>  e.g.,  Nil  &  NAVAIR  on  Information  Assurance  (Action  1-6) 

6.  POM  initiative  (none  so  far,  but  under  discussion) 
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Some  Recent  Funding  Successes  <i  of  4) 

□  Successfully  competed  for  M&S  SC  funds  for  these  projects, 
currently  underway  with  SSE/DT&E  oversight 

>  07-1  -001  f  Integrated  Natural  Environment  Authoritative 
Representation  Process  (AMSMP  Action  4-4d) 

Deliverable:  Environmental  Scenario  Generator  that  provides  better  and 
more  rapid  generation  of  weather,  space,  and  terrain  representations 

Program  Manager:  Col  Mark  Zettlemoyer,  USAF  (MSCA) 

Performer:  SAIC 

$2.3M 

>  07-1  -002f  M&S  Resource  Reuse  Business  Model  (AMSMP  Action 
4-3a) 

Deliverable:  Recommended  business  model  (including  policy,  incentive 
structure,  and  procedures)  for  the  reuse  of  M&S  resources  and  a  campaign 
plan  for  implementation 

Program  Manager:  Mr.  Chris  DiPetto  (was  Lt  Col  White) 

Performer:  Center  for  Naval  Analysis  (Dr.  Dennis  Shea,  et.  al.) 

$800K 
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Some  Recent  Funding  Successes  <2  of 4> 


>  07-1  -004f  Educating  the  M&S  Workforce  (AMSMP  Actions  5-1  and  5-3) 
Deliverables: 

-  Required  workforce  M&S  competencies 

-  Learning  architecture  to  define  content,  instructional  delivery  methods,  and 
scope 

Program  Manager:  ODASN(RDA)/CHENG  (W.  Zimmerman) 

Performer:  Naval  Postgraduate  School,  other  academic  partners, 

$3.2M 

>  07-1 -005f  W&A  Standardization  (AMSMP  Action  4-5a) 

Deliverables: 

1 .  VV&A  standardized  documentation  template 

2.  VV&A  documentation  tool  to  assist  users 

3.  Policy  and  guidance  updates 

PM:  Director,  Navy  Modeling  and  Simulation  Office  (K.  Charlow) 

Performer:  SPAWAR 
$550K 
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Some  Recent  Funding  Successes  <3  of 4> 


>  060-TR-01  Live  Virtual  Constructive  Architecture  Roadmap  (AMSMP 

Actions  2-3  and  3-3a) 

Deliverables: 

-  Functional  requirements  for  Live-Virtual-Constructive  simulation  environments 

-  Capabilities  &  limitations  of  various  distributed  simulation  architectures  in  use 

across  DoD  (DIS,  ALSP,  HLA,  TENA,  CTIA) 

-  Comparative  analyses  of  the  architectures,  middleware,  business  models,  and 

standards  management 

-  Analysis  of  alternatives 

-  Recommended  roadmap 

Oversight:  P&R  and  DUSD  (A&T)/SSE/DT&E 
Program  Manager:  JFCOM  (Mr.  Ken  Goad) 

Performer:  JFCOM,  IDA,  JHU  APL,  PEO-STRI 
$1.4M 
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Some  Recent  Funding  Successes  <4  of  4) 

Successfully  competed  for  OSD  Study  Funds  for: 

>  Study  on  Best  Practices  for  M&S  Tool  Development 

(AMSMP  Action  3-2) 

Deliverables: 

■  Bibliography  identifying  sound  practices 

■  Draft  and  final  version  of  best  practices  for  M&S  tool  development 
Program  Manager:  Col  Sean  McAllum,  USF,  ODUSD(A&T)/SSE/DT&E 
Performer:  JHU  APL 

$350K 

>  Study  on  Management  of  Broadly-needed  M&S  tools 

(AMSMP  Action  3-4) 

Deliverables: 

■  Best  practices  for  managing  broadly  needed  M&S  tools 

■  Recommended  actions  to  improve  DoD  management  of  such  tools 

Program  Manager:  Col  Sean  McAllum,  USF,  ODUSD(A&T)/SSE/DT&E 

Performer:  JHU  APL 

$500K 


Objective  1 


Provide 
necessary 
policy  and 
guidance 


1-1  M&S 

management 


Execution  Progress  Overview 

Objective  3 


Improve 
model  and 
simulation 
capabilities 


Objective  2 


Enhance  the 
technical 
framework 
for  M&S 


Objective  4 


Improve 
model  and 
simulation 
use 


2-1  Product 
development 
metamodel 

2-2  Commercial 
SE  standards 

2-3  Distributed 
simulation 
standards 

2-4  DoDAF  utility 

Si^DoDAF  2.0 
S^t^ms 
Engim*feiqng 
Overlay 

b)  Standards  for 
depiction  & 
interchange 

2-5  Metadata 
template  for 
reusable 
resources 


3-2  Best  practices 
for  model/sim 

development  4-3  Foster  reuse 


3-4  Central  funding 
of  high-priority, 
broadly-needed 
models  &  sims 

a)  Prioritize  needs 


4-4  Info  availability 

b)  Systems 

c)  Threats 

4-5  W&A 
a)  Documentation 


Objective  5 


Shape  the 
workforce 


5-1  Definition  of 
required  M&S 
competencies 

5-2  Harvesting  of 
commercial 
M&S  lessons 

5-3  Assemble  Body 
of  Knowledge 
for  Acqn  M&S 

5-4  M&S  education 
&  training 
a)  DAU,  & 
on-line  CLMs 
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Outline 


□  AMSMP  Development  (Review) 

□  AMSMP  Execution 

>  Funding  approach 

>  Progress  overview 

□  Future  Plans 

□  Q&A/Status  of  Individual  Actions 


Future  Plans  (FY09/10) 

□  Continue  cooperatively  executing  the  AMSMP 

□  Update  AMSMP  to  reflect  accomplishments,  fact  of  life 
changes,  and  newly-identified  needs  (e.g,  Virtual  Battlespace 
Center  for  OSD  acqn  decisions).  Make  vision  more  specific. 

□  Ensure  programs  know  about  and  can  access  deliverables 

□  Provide  direct  assistance  to  programs 

>  At  the  request  of  SSE/Assessment  and  Support,  have  already 
conducted  M&S  review  of  Joint  Light  Tactical  Vehicle  and  FCS 

□  Continue  to  educate  and  learn  via  outreach 

>  Conferences  and  workshops,  both  defense  &  commercial 

□  Support  development  of  useful  standards 

>  SISO,  W3C  Data  Semantics  WG,  OMG,  etc. 

□  Pursue  additional  resources  (both  people  and  $) 
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Outline 


□  AMSMP  Development  (Review) 

□  AMSMP  Execution 

>  Funding  approach 

>  Progress  overview 

□  Future  Plans 

□  Q&A/Status  of  Individual  Actions 

>  Will  gladly  discuss  individual 
actions  of  interest 


AMSMP 


Progress  Overview 


Objective  1 


Provide 
necessary 
policy  and 
guidance 


1-1  M&S 

management 


Separate  presentation 


Objective  2 


Enhance  the 
technical 
framework 
for  M&S 


Objective  3 


Improve 
model  and 
simulation 
capabilities 


Objective  4 


Improve 
model  and 
simulation 
use 


Objective  5 


Shape  the 
workforce 


2-1  Product 
development 
metamodel 

\ 

2-2  Commercial 
SE  standards  , 

2-3  Distributed 
simulation 
standards 

2-4  DoDAF  utility 

Si^DoDAF  2.0 
S^t^ms 
Engim*feiqng 
Overlay 

b)  Standards  for 
depiction  & 
interchange 

2-5  Metadata 
template  for 
reusable 
resources 


3-2  Best  practices 
for  model/sim 
development 


3-4  Central  funding 
of  high-priority, 
broadly-needed 
models  &  sims 

a)  Prioritize  needs 


4-3  Foster  reuse 


4-4  Info  availability 

b)  Systems 

c)  Threats 


5-1  Definition  of 
required  M&S 
competencies 

5-2  Harvesting  of 
commercial 
M&S  lessons 

5-3  Assemble  Body 
of  Knowledge 
for  Acqn  M&S 

5-4  M&S  education 
&  training 
a)  DAU,  & 
on-line  CLMs 


4-5  VV&A _ 

a)  Documentation 
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Status  of  Individual  Actions 


Caveat:  Did  not  rate  down  progress  for 
lateness,  unless  stalled 
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Objective  1 :  Provide  Necessary  Policy  &  Guidance 


1-1.  Provide  effective,  persistent  DoD-wide  M&S  management  to  address 
cross-cutting  M&S  issues,  coordinate  actions 

Lead:  OUSD(AT&L)  Support:  OUSD(AT&L)/DS(SSE),  OUSD(P&R),  OUSD(C)/PA&E,  etc. 

Products:  Revised  DoDD  5000.59  (M&S  Management),  revised  senior  leadership 
management;  and  improved  policies  for  M&S  management,  revised  senior  leadership 
management;  and  improved  policies  for  M&S  management. 

Completion  goal:  2006 


j  Now  DoD  M&B  management  airueiujre  In  place;  effectiveness  questioned 

j  New  DoD  Directive  finaJJy  released  Aug  07,  with  promise  of  a  follow-on  DoDJ  i :o 
define  key  responsibilities  and  processes,  DOP  now  proposed  as  substitute, 

•  No  acquisition  community  leadership  role  on  M&S  SC  (Training  &  Analysis  do) 

•  Current  project  selection  process  does  not  fund  only  cross-cutting  efforts, 
misusing  M&S  PE 


Next  Steps: 

•  Continue  to  argue  for  an  SSE  leadership  role  on  M&S  SC 

•  Advocate  within  M&S  governance  structure  for  a  DoDI  on  M&S  management 

•  Continue  to  propose  an  alternative  approach  to  “C&CC  Business  Plan” 
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Objective  1 :  Provide  Necessary  Policy  &  Guidance 


1-2.  Promote  model-based  systems  engineering  (MBSE)  and  M&S-enabled 
collaborative  environments,  at  both  the  program  and  joint  capability  level 

Lead:  OUSD(AT&L)/DS(SSE);  Support:  Components 
Products:  Revised  guidance  in  DAG 

Completion  goal:  2007 


•  Current  DAG  mentions  coJJ aboratlve  environments  14  times, 
simulation-based  testing  once,  SDA  twice,  and  MDGJI  not  at  aJL 

j  Programs/companies  often  claim  collaborative  environments,  but  only  partial 

•  MBSE  a  prominent  part  of  INCOSE’s  CM  Vision  2020 
j  Increasing  industry  use  of  M«3SE  concept  &  tools 

•  GDI  submitted  new  DAG  language  May  07,  but  DAG  revision  stalled 


Next  steps: 

•  Continue  advocacy  for  submitted  DAG  language;  revise  submittal  if  rejected 

•  Investigate  possibility  of  a  CLM  on  MBSE 
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Objective  1 :  Provide  Necessary  Policy  &  Guidance 


1-3.  Establish  policy  and  guidance  on  appropriate  use  of  M&S  to  plan  tests,  to 
complement  system  live  tests,  and  to  evaluate  joint  capabilities 

Co-leads:  OUSD(AT&L)/DS,  ODOT&E;  Support:  Components 
Products:  Revised  policy  and  guidance  in  DoDI  5000.2  and  DAG 

Completion  goal:  2007 


•  DoDJ  3000,2  is  excellent  at  the  program  level,  b ut  not  at  the  capability  JeveJ 
j  better  discussion  in  bbE's  latest  I) AG  submission,  but  need  more  specificity 
j  JMITC  launched,  but  many  challenges  ahead,  including  policy 
j  Services  are  getting  more  active  (e,g,,  NAVAIR  Enterprise  Initiative) 


Next  steps: 

•  NDIA  M&S  Cmte  participate  in  DT&E  Cmte  effort;  check  for  progress 

•  Track  JMETC  policy  development,  respond  appropriately 

•  Continue  working  with  NAVAIR  M&S  Enterprise  to  develop  guidance 

•  Draft  expanded  policy  and  guidance,  vet  with  the  various  stakeholders 

•  Submit  additional  changes  to  DAG  (both  T&E  and  M&S  portions) 
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Obj.  1 :  Provide  Necessary  Policy  &  Guidance  (cont.) 


1-4.  Establish  policy  to  require  documented  M&S  planning  at  the  joint 
capability  &  program  levels  as  part  of  the  Systems  Engineering  Plan, 
T&E  Strategy  and  T&E  Master  Plan 

Co-leads:  OUSD(AT&L)/DS(SSE),  ODOT&E;  Support:  Components 
Products:  Revised  policy  and  guidance  in  DoDI  5000.2,  DAG,  and  DOT&E  TEMP 
Planning  Guidance 

Completion  goal:  2007 


j  AbKJWG  (DDI)  submitted  revised  language  to  DoD  3000,2,  DAG  Janguage  and 
SEP  Preparation  Guide 

>  Par  dal  acceptance  of  DIP  language;  DoDI  5000.2  and  DAG  updates  stalled 

>  Mo  action  thuo  far  regarding  flblP  Planning  Guidance 


Next  steps: 

•  Continue  working  with  NAVAIR  M&S  Enterprise  to  develop  guidance 

•  Draft/submit  language  for  TEMP  Planning  Guidance 
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Obj.  1 :  Provide  Necessary  Policy  &  Guidance  (cont.) 


1-5.  Establish  M&S-related  guidelines  for  solicitations,  source  selections, 
and  contracting. 

Lead:  OUSD(AT&L)/DS(SSE);  Support:  OUSD(AT&L)/DPAP,  ODOT&E,  Components 
Products:  Sample  language  in  DoD  publications  (e.g.,  DAG,  SEP  Preparation  Guide, 
Contracting  for  Systems  Engineering  Guidebook)  regarding  M&S  requirements,  data 
rights,  and  the  responsibilities  and  liabilities  of  parties  regarding  sharing  and  reuse 

Completion  goal:  2007 


Solicited  inputs  fro//)  AbJbWG  /no/nboro  ond  industry  (through  DDJA  bJ&S 
Crnto) 

j  AhlbWG  (bbEi)  submitted  DAG  language  regarding  aouree-eeieetion  criteria 
>  Presentation  at  Oct  0/  MDIA  by  sterns  Engineering  Conference 

•  Action  completion  is  overdue  (2007)  due  to  M&S  Cell  resource  constraints 


Next  steps: 

•  Further  refinement  and  vetting  of  proposed  guidance 

•  Synthesize  best  language  &  submit  to  DAG  (update),  SEP  Preparation  Guide, 
and  Contracting  for  Systems  Engineering  Guidebook 
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Obj.  1 :  Provide  Necessary  Policy  &  Guidance  (cont.) 


1-6.  Ensure  practical  guidelines  for  information  assurance  certification 
and  accreditation  of  M&S  federated  networks  falling  under  multiple 
Designated  Accreditation  Authorities  (DAAs) 

Lead:  OASD(NII);  Support:  OUSD(AT&L)/DS(SSE),  OUSD(I),  NSA 
Products:  Proven,  practical  guidelines  published  in  DAG  and  DoD  8500. 2-H,  per 
DoDI  8500.2  “Information  Assurance  Implementation,”  Feb  6,  2003 

Completion  goal:  2007 


•  Nil  has  published  DoDJ  3300.2,  but  AMDWG  guesbons  adequacy 

j  AMSWO-NU  discussions  helcJ  in  200 7;  MA/AIR  procedures  itl entitled  as  a 
candidal  to  provide  the  additional  specificity  needed 

j  Awaiting  delivery  of  NAVAJD  procedures  for  (a)  NJJ  evaluation  of  compliance 
with  3500.2.  (b)  jnJJI  evaluation  of  suitability  for  revising  3300.2,  and  (e)  AM8WG 
evaluation  of  suitability  for  inclusion  in  DAG 


Next  steps: 

•  Follow-up  with  NAVAIR  to  ensure  submission  of  their  procedures 

•  Conduct  three  evaluations  mentioned  above 

•  Draft,  vet,  and  submit  DAG  language 
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Objective  2:  Enhance  the  Technical  Framework  for  M&S 


2-1 .  Develop  a  product  development  information  metamodel  &  associated 

metadata  extensions  to  the  DoD  Discovery  Metadata  Specification 
Lead:  OUSD(AT&L)/DS(SSE);  Support:  OASD(NII),  Components 
Products:  Revised  DDMS;  revised  guidance  in  DAG. 

Completion  goal:  2008 


j  -J-jF  bcj'j  developed  o  /notaniodoJ  Epocificntlon  and  provided  it  to  IVJ&S  CO 

j  Wo  requested,  sin d  CO  provided,  Cerudder  existence  to  work  with 
JCk  to  evolve/reflne  Jto  rnetsiniodel 

j  Working  group  hoo  decided  key  ioouas  end  expects  to  public!)  o  revised 
version  shortly 


Next  steps: 

•  JSF  to  complete  revised  metadata  specification 

•  Coordinate  with  M&S  CO  to  vet  more  broadly  (likely  PA&E  interest)  and 
make  this  a  DoD  or  (preferably)  commercial  standard 

•  Submit  into  DoD  Standardization  Program  process 
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Objective  2:  Enhance  the  Technical  Framework  for  M&S 


2-2.  Support  development  of  open  commercial  and  non-proprietary 
standards  for  (model-based)  systems  engineering,  such  as  OMG’s 
Systems  Modeling  Language  (SysML)  and  ISO  Standard  10303  AP- 
233 


Co-leads:  OUSD(AT&L)/DS(SSE);  DoD  CIO  Support:  OASD(NII),  DLA, 
OUSD(AT&L),  Products:  Standards  suitable  for  use  by  DoD 


•  Action  jo  complete  for  byoblL  end  A  P-2 -33,  but  DoD  oworeness  jo  Joclbng 

•  SysML  vLQ  issued  no  on  “ovoiJobJe  stojfidord;”  v  3/J  minor  revision  Jote  2003 
->  Jncreosing  usoge  &  teoching  of  DysbJL;  mojor  subject  ot  Jj1<DOSIE,  NIDJA 

•  Novy  bLAb  Dtondords  Steering  Croup  boo  proposed  DysblL  as  a  stamlord 
j  AP-233  DP  do  to  interchonge  stondords  being  reieosed  incrementolly 

>  CO  TD  bye  tom  Engineering  tools  ore  incorporating  DysblL  ond  AP  -233 
j  Nothing  yet  submitted  to  DoD  Dtondordsotion  Program  ond  DibR 


Next  steps: 

•  Track  SysML  and  AP-233  implementations,  publicize  results 

•  Investigate  DoD  Standardization  Program  process;  submit  SysML  and  AP-233 

•  Identify  any  needs  for  additional  standards 
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Objective  2:  Enhance  the  Technical  Framework  for  M&S 


2-3.  Establish  a  forum  to  clarify  the  characteristics  and  application  of 
various  distributed  simulation  standards  (ALSP,  DIS,  HLA,  SI3,  TENA, 
etc.)  and  examine  opportunities  for  convergence 

Lead:  OUSD(AT&L)  Support:  OUSD(AT&L)/TRMC  &  DS(SSE),  ODOT&E, 
Components 

Products:  (1 )  Information  on  strengths  &  weaknesses  of  the  various  standards;  (2) 
agreement  on  policy  and/or  guidance  on  the  use  of  distributed  simulation  standards; 
(3)  a  way  ahead  regarding  distributed  simulation  standards 

>  Completion  goal:  2007 


j  'jC-'fimded  LVC  Afchiteciure  fiosid/jrmp  in  2007,  du9  o ui:  Jni:9  2000 

->  iy  interested,  hns  teken  on  9  briefing 

•  1VI&6  Coll  (Mollenbach)  participating  in  diiy  project,  tracking  progress  and 
coordinating  related  MSS  3C  actions  (Ml. A  Way  Ahead) 


Next  steps: 

•  Continue  to  participate;  await  final  report 

•  Help  shape  M&S  SC  response 
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Obj.  2:  Enhance  the  Technical  Framework  for  M&S  (cont.) 


2-4.  Improve  the  utility  of  the  DoD  Architecture  Framework  (DoDAF)  for 
acquisition 

2-4(a)  Develop  Systems  Engineering  Overlay  (mofWe)  for  DoDAF  v2.0 

Lead:  O^D(AT&L)/DSj/oupport:  OAS^(NII),  Comporamts  Y 

Products:  Acquisition  ioverlay  for  DoDAF  v2.0  /  / 

Completion  goal:  2p06  /  /  / 


2-4(b)  Support  development  of  open  commercial  standards  for  the 
depiction  and  interchange  of  DoDAF-compliant  architectures 

Lead:  OASD(NII)  Support:  OUSD(AT&L)/DS(SSE) 

Products:  Published  standards  suitable  for  adoption  by  DoD  in  DoDAF  2.0;  revised 
guidance  in  DAG 

Completion  goal:  2007 


•  2-4(a):  DoDAF  Overlay  concept  has  been  dropped,  so  this  action  is  OBE 

^  2A[b):  Oi'JlG's  UPDi'Ji  (UML  Profile  for  DoDAFMODAF)  nearly  finalised,  MW 
hue  embraced  UPDj'jJ  as  an  element  of  DoDAF  2.0  develop/neni: 

j  GC  Forum  considering  die  vaiue  an d  impact  of  DoDAF 

j  AGD(NJJ)  is  attempting  to  make  DoDAF  v2.0  more  useful  for  acquisition 

j  Acquisition  Community  participation  in  DoDAF  WG  curtailed 

Next  steps: 

•  Increase  involvement  in  DoDAF  WG 

•  Submit  UPDM  to  DoD  Standardization  Program  I DISR  Online 

•  Advocate  use  of  UPDii  for  architecture  data  exchange _ 
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Obj.  2:  Enhance  the  Technical  Framework  for  M&S  (cont.) 


2-5.  Establish  a  standard  template  of  key  characteristics  (metadata)  to 
describe  (discover)  reusable  M&S  resources 

Lead:  OUSD(AT&L)  Support:  OUSD(AT&L)/DS(SSE)  &  TRMC,  OASD(NII), 
ODOT&E,  Components 

Products:  Published  standard  template;  usage  guidance  in  DAG 

Completion  goal:  2007 


j  i'MC  CO  bKVJ  COJ  Discovery  Metadata  project  addressee  tb jo 

•  i'MO  Cell  has  coordinated  with  M&B  CO  to  ensure  no  cross-threads  with 
Action  2-1  (Product  Development  Metadata  Specification) 

•  Version  1.0  published,  being  evaluated  by  users  (e.g.,  IMCRR,  JDC,  JPSG) 
who  ere  providing  feedback  to  refine  it 


Next  steps: 

•  Draft,  vet  and  submit  DAG  entry  when  final  product  is  available 
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Objective  3:  Improve  Model  &  Simulation  Capabilities 


3-1 .  Establish  a  process  to  ensure  acquisition  needs  are  reflected  in  DoD 
M&S  priorities 

Lead:  OUSD(AT&L)  Support:  OUSD(AT&L)/DS(SSE),  ODOT&E,  DOD  CIO, 
Components 

Products:  A  method  to  capture  and  prioritize  acquisition  needs. 

Completion  goal:  2007 


•  AiVloWG  ha  a  auccoaolully  obtained  CC  funding  for  several  projects 

*  AMBWCj  ha  a  started  an  effort  to  pursue  SRJR  opportunities 

>  AM  43  WO  till  doea  not  have  an  effective  voice  in  other  venues  that  affect  jVJ&O 
capability,  such  a  a  other  43  &7  and  DARPA 


Next  steps: 

•  Continue  to  pursue  M&S  SC  and  SBIR  funding  opportunities 

•  Investigate  DoD  S&T  planning  process  to  identify  entry  points 

•  Build  list  of  acquisition  M&S  S&T  needs 
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Objective  3:  Improve  Model  &  Simulation  Capabilities 


3-2.  Define  and  foster  best  practices  for  efficient  development  and 
evolution  of  credible  M&S  tools,  incorporating  user-defined 
requirements,  a  systems  engineering  approach,  and  appropriate 
verification  &  validation 

Lead:  OUSD(AT&L);  Support:  OUSD(AT&L)/DS(SSE),  ODOT&E,  DOD  CIO, 
Components 

Products:  Best  practices  publication,  available  via  MSIAC,  DTIC,  etc.;  DAG  guidance 
to  use 

Completion  goal:  2008 


j  rk ive  obtained  OBD  study  funds  lorihd  definition  poirtion  of  this  tusk 
j  SOW  written 

5  Contracting  with  JHU  APL  to  develop  this  best  practice 


Next  steps: 

•  Assess  JHU  APL  deliverable 

•  Foster  its  use  (via  Action  5-4) 
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Obj  3:  Improve  Model  &  Simulation  Capabilities  (cont.) 


3-3.  Enable  readily-available  distributed  live-virtual-constructive  environments, 
leveraging  related  initiatives 

3-3(a)  Establish  DoD-wide  standards  for  distributed  environments 

Lead:  OUSD(AT&L);  Support:  OUSD(AT&L)/TRMC  &  DS(SSE);  ODOT&E;  DOD  CIO,  Components 
Products:  Published  standard;  DODI  (#  TBD)  policy  to  use 

Completion  goal:  2008 

3-3(b)  Make  candidate  simulations,  labs  and  ranges  compliant  with  these 
standards 

Lead:  Components;  Support:  OUSD(AT&L)/DS(SSE)  &  TRMC,  ODOT&E 
Products:  A  larger  collection  of  simulations,  labs,  and  ranges  ready  to  be  employed  in  distributed 
events 

Completion  goal:  2010 


3-3(c)  Ensure  availability  of  services  to  help  plan  and  conduct  events 

Lead:  Components;  Support:  OUSD(AT&L),  OUSD(AT&L)/TRMC,  DISA 

Products:  Fee-based  technical  services  to  help  users  (e.g.,  PMs,  Capability  Managers,  OTAs)  plan 
and  conduct  distributed  events 

Completion  goal:  2009 


j  LVC  Architecture  Boadmap  and  Jr  Joint  Co/npooabJe  Object  Model 
projects  underway 

j  Virtual  OattJeopace  Center  Defence  Science  Board  Tack  Ferae  in  work 
j  Mo  funding  yet  available  to  do  trie  root 


Next  steps: 

•  Await  LVC  Architecture  Roadmap,  support  implementation  as  appropriate 

•  Pursue  POM  initiative 
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Obj  3:  Improve  Model  &  Simulation  Capabilities  (cont.) 


3-4.  Centrally  fund  and  manage  the  development  of  high-priority,  broadly- 
needed  M&S  tools 


3-4(a)  Identify  and  prioritize  broadly-needed  M&S  tools 

Lead:  OUSD(AT&L);  Support:  OUSD(AT&L)/(SSE);  ODOT&E,  DOD  CIO,  Components 
Products:  Prioritized  list  of  common  M&S  tool  needs 

Completion  goal:  2007 

3-4(b)  Conduct  one  or  more  pilot  projects  to  develop  new  M&S  tools  or 
update  existing  ones  to  meet  these  needs 

Lead:  OUSD(AT&L);  Support:  OUSD(AT&L)/DS(SSE),  Components 
Products:  Proof  of  concept  for  managing  the  development/evolution  of  M&S  tools  to 
meet  broadly-shared  needs 

Completion  goal:  2008 


3-4(c)  Expand  the  scope  of  central  M&S  tool  management  as  warranted 
by  pilot  project  results  and  the  list  of  common  M&S  needs 


Lead:  OUSD(AT&L);  Support:  OUSD(AT&L)/DS(SSE),  ODOT&E,  Components 
Products:  Capability  to  provide  broadly-needed  M&S  tools  in  a  more  responsive  and 
cost-effective  way. 

Completion  goal:  2011 


J  AMoWO  submitted  3-4 (b)  pjJot  propo^J  to  bJib  bC,  but  it  wasn't  funded 

j  Funding  obtained  to  have  JHU  AP L  identify  bee t  praetieee  for  managing 
broadly  needed  3  toola  an d  recommend  i)ol)  actions 

Next  steps: 

•  Assess  JHU  APL  deliverables,  pursue  actions  as  appropriate 
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Objective  4:  Improve  Model  &  Simulation  Use 


4-1.  Provide  potential  acquisition  M&S  users  the  knowledge  needed  to 
formulate  an  effective  M&S  strategy  via  ready  access  to  M&S  expertise 
and  information  about  M&S  capabilities  and  gaps,  reusable  resources, 
lessons-learned,  etc. 

Lead:  OUSD(AT&L);  Support:  OUSD(AT&L)/DS(SSE) 

Products:  Revised  guidance  in  DAG;  improved  knowledge  base  in  MSIAC;  assist  visits 
(e.g.,  by  OUSD(AT&L)/DS(SSE) 

Completion  goal:  2008 


*  rtevised  guidance  submitted  to  DAG 

*  BCD  Cell  assisting  as  able,  but  resource  limited,  not  widely  advertised 
j  Navy  coming  on  line,  but  no  notion  from  other  Components 

>  6 ■  !  Education  project  Identified  M&S  Bodies  of  Knowledge  that  offer  useful 


information 


Next  steps: 

•  Advertise  and  expand  assist  visits.  SSE  has  made  this  a  2008  priority. 

•  Based  on  our  experience,  promote  similar  efforts  by  other  Components 

•  Improve  MSIAC  expertise  regarding  M&S  in  acquisition  (Action  5-5) 
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Objective  4:  Improve  Model  &  Simulation  Use 


4-2.  Define  and  disseminate  best  practices  for  disciplined  M&S  planning  & 
employment 

Lead:  OUSD(AT&L)/DS(SSE),  Support:  OUSD(AT&L),  Components 
Product:  Revised  best  practices  guidance  in  DAG  and  MSIAC 

Completion  goal:  2007 


•  High-level  discussion  included  in  “M&S  for  Systems  Engineering"  CLIbl 

•  Expended  discussion  submitted  in  recent  DAG  revision 

•  bl&D  Planning  end  Employment  Dest  Practices  solicitation  completed  Apr  07 

•  MAVA1R  iVI&D  Enterprise  is  developing  recommendations 

•  Action  completion  is  overdue  (2007)  due  to  M&S  Cell  resource  constraints 


Next  steps: 

•  Continue  working  with  NAVAIR  M&S  Enterprise  to  develop  guidance 

•  Synthesize  best  practice,  conduct  AMSWG  &  NDIA  reviews 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 


4-3.  Facilitate  the  sharing  of  reusable  resources 

4-3(a)  Establish  a  DoD-wide  business  model  for  compensating  providers 
of  reusable  M&S  resources  (e.g.,  information,  software,  services) 

Lead:  OUSD(AT&L);  Support:  OUSD(AT&L)/DS(SSE),  OUSD(P&R),  OUSD(C)/PA&E, 
Components 

Product:  Documented  business  model;  revised  policy  and/or  guidance  in  DoD  5000  series 
&  DAG 

Completion  goal:  2007 


•  3C-rundyd  ikyyourcy  iky uyy  -duyjnyyy  bJodyJ  siudy  umJyrwyy,  wjJJ 

rypori  out  20 0B 

j  odidy  will  identify  kyy  jyyuyy  and  recommend  significant  cbyngas 
3  LVCAik  will  yJ-30  eddry33  business  rnodei  j^ujos 


An  effective 


model  is  not  yet  established 


Next  steps: 

•  No  further  action  needed  yet;  awaiting  study  outcome 

•  LVC  Architecture  Roadmap  may  have  an  impact 

•  Take  action  to  implement  study  &  LVCAR  recommendations  as  appropriate 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 


4-3.  Facilitate  the  sharing  of  reusable  resources 


4-3(b)  Establish  DoD  policy  and/or  guidance  regarding  responsibilities 

to  share,  protect  and  properly  use  M&S  information,  tools,  and  data 
Co-Leads:  OASD(NII),  OUSD(AT&L),  USD(I);  Support:  OUSD(AT&L)/DS(SSE)  & 
j  DPAP,  OUSD(P&R),  OUSD(C)/PA&E,  Components 
Product:  Revised  policy  and/or  guidance  in  various  issuances  (e.g.,  DoD  5000  series, 
DAG,  contracting  guidance) 

Completion  goal:  2008 


•  Drafted  and  submitted  DAG  language,  but  not  yet  included  in  DAG 

#  fteaouroe  Jkeuae  Duaineoa  1'jJodeJ  project  /nay  make  recommendation: 
on  thio  subject 


Next  steps: 

•  Receive  Business  Model  study  report,  take  action  as  appropriate 

•  Draft  language  for  contracting  guide 

•  (DODI  5000.2  change  may  not  be  needed) 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 

4-3.  Facilitate  the  sharing  of  reusable  resources 


4-3(c)  Enhance  the  means  (e.g.,  directory  service,  registries,  bulletin 
ooards)  to  discover  the  existence  of  reusable  resources  required  for 


M&S  and  contact  information 

Lead:  OUSD(AT&L)  Support:  OUSD(AT&L)/DS(SSE),  OUSD(P&R),  OUSD(C)/PA&E, 
Components 

Product:  A  better  way  to  discover  reusable  resources.  Re-orientation  and  integration  of 
various  DoD  M&S  resources  repositories. 

Completion  goal:  2007 


•  DDrt&£-direeted  M&S  CO  project  to  develop  a  “M&C  Resource  Co  to  Jog"  jo 
undervyay 

>  We  aee  a  viable  buaineaa  model  aa  a  prerequisite 


Next  steps: 

•  Track  M&S  CO  project,  support  as  able 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 


4-4.  Define  the  types  of  information  DoD  organizations  shall  make  available  to 
others  with  a  clearance  and  valid  need  to  know  and  the  processes  to  obtain 
them  (per  reuse  business  model).  The  process  to  obtain  information  should 
include  an  efficient  mechanism  for  industry  to  request  government  data  with 
specific  "need  to  know"  outside  a  specific  contract  environment. 

4-4(a)  Scenario  data 

Lead:  OUSD(AT&L)  Support:  OCJCS(J8),  OUSD(C)/PA&E,  DIA,  Components 
Product:  Approved  scenarios  and  process  to  obtain 

Completion  goal:  2007 

4-4(b)  System-related  data 

Lead:  OUSD(AT&L)/DS(SSE);  Support:  ODOT&E,  Components 
Product:  Process  to  obtain  authoritative  system  data  (characteristics  and  performance, 
interactions,  interfaces,  logistic  support,  etc.)  documented  in  the  DAG  and  appropriate 
OASD  (Nil)  policy  documents. 

Completion  goal:  2008 

4-4(c)  Threat  data 

Lead:  DIA;  Support:  OUSD(AT&L);  OUSD(AT&L)/DS(SSE),  ODOT&E,  and 
Components 

Product:  Authoritative  threat  data  and  process  to  obtain 

Completion  goal:  2007 

4-4(d)  Natural  environment  data 

Lead:  DoD  Natural  Environment  MSEAs  (MSCAs);  Support:  OUSD(AT&L), 
OUSD(AT&L)/DS(SSE),  Components 

Product:  Authoritative  natural  environment  data  and  process  to  obtain 

Completion  goal:  2007 


44 


Action  4-4  Assessment 


J  i 


Vjtjujsldon  Support  Division  of  Dl;\  boo  briefed  AIjJSWG  and  NDIA  M&S  Cmte 
on  Ite  support  lo  oocjiiijojbon  programs 

j  M8JC  boo  briefed  MDJA  i'iJiS  Crnce  on  ‘fMAP  program  end  provided 
Instructions  on  bow  to  regueslTMAP  models 

j  Drufi:  DAG  language  dioouooeo  threat  data  sources  and  traceability 

j  Mo  method  exists  “for  industry  to  request  government  date)  with  specific 
biood  co  know'  outside  a  specific  contract  environment” 

1  SC-funded  Invironrnental  Scenario  Generator  project  underway 

•  No  progress  in  sharing  U.S.  system  data 

>  Joint  iterpid  Scenario  Generation  (JDSG)  and  Joint  Data  Alternatives  (JDA) 
projects  advertise  they  will  address  oil  the  Action  4  A  info  needs;  time  will  coll 


Next  steps: 

•  Monitor  JRSG  and  JDA  projects  as  resources  permit 

•  Investigate  data  sharing  polices  of  OSD,  JCS,  and  other  Components 

•  Investigate  JSC,  PAE,  &  Service  scenario  data  availability  &  access 

•  Draft  additional  DAG  language  on  all  data  types  (interim  prior  to  JRSG  /JDA) 

•  Continue  to  build  on  Nov  07  PA&E-Boeing-NDIA  M&S  Cmte  discussion 

•  Examine  benefits  of  establishing  a  DoD  Virtual  Battlespace  Center 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 


4-5.  Foster  cost-effective  VV&A 

4-5(a)  Require  DoD-wide  standardized  documentation  of  VV&A 

Lead:  OUSD(AT&L);  Support:  OUSD(AT&L)/DS(SSE),  ODOT&E, 
Components 

Products:  Revised  policy  in  DODI  5000.2  and  5000.61;  revised  guidance  in 
DAG 

Completion  goal:  2007 


•  Ai'jJbWG^oponaorod,  CcAfundod  project  co/n|p]oi:od 

•  Doeu/noniadon  hi  vs  boon  ootobJjohod  no  a  jVJJL3P^C  3022;  co/mnorcjal 
(0J0O)  Candard  to  follovy 

•  Tool  to  rnanago  docurnontotion  jo  in  bo  to  tootinp 

>  A  M3  WO  con  corn  that  draft  M&3  3  Co  “DoD  M&3  btratogio  Vision"  oaJI  for 

“practical  verification,  validation,  and  accreditation  guidelines  that  vary  by 
application  area”  (o/nphaob  addod)  vvlIJ  undor/nino  VV&A, 

•  PA&E  resisting  this  requirement  in  DoDI  5000.61  revision 

Next  steps: 

•  Publicize  standard  and  supporting  tool 

•  Fight  to  have  DoDI  5000.61  to  maintain  a  consistent  DoD  policy  and  require 
documentation  per  MILSPEC 

•  Establish  a  commercial  standard  under  SISO 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 


4-5.  Foster  cost-effective  VV&A 


4-5(b)  Develop  risk-based  methodology  and  associated  guidelines  for 
VV&A  expenditures 

T  Lead:  OUSD(AT&L);  Support:  OUSD(AT&L)/DS(SSE),  Components 

Products:  Updated  DoDI  5000.61;  revised  policy  and  guidance  in  DoDI  5000.2 
and  DAG 

Completion  goal:  2007 


M2&  CO  project  underway,  with  promise  H  wjJJ  iiddr^S'iJim  issua 

NAVA  If*  iC  ^.ndjf'pf'ioO  developing  iVJ&C  VV&A  end  riok  menaporrioru 

guidance 


Next  steps: 

•  Assess  M&S  Enterprise  guidance 

•  Obtain  update  on  M&S  CO  progress  developing  risk-based  W&A 
guidelines,  support  and  take  action  as  necessary 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 


4-5.  Foster  cost-effective  VV&A 


4-5(c)  Examine  a  program’s  VV&A  when  M&S  informs  major  acquisition 
decisions  and  unambiguously  state  the  purpose,  key  assumptions  and 
significant  limitations  of  each  model/simulation  when  results  are  presented. 

ij  Lead:  OUSD(AT&L)/DS(SSE)  Support:  DoD  Components 

Products:  Guidance  &  training  for  oversight  personnel;  updates  to  DAG  Chaps  4,  9 

Completion  goal:  2007 


•  Bubnmted  DAG  language  on  VV&A  ^nirnimidon,  but  DAG  update  stalled 

•  SSE  IM&S  Cell  Inis  gjtmn  initial  bribing  to  OUSD(A&T)/SSE/AS 

>  Navy  may  bo  addressing  diis;  no  odior  Conrpononi  aodvidos  underway 


Next  steps: 

•  Broaden  teaching  on  VV&A  examination 

•  M&S  Cell  support  SSE/AS  to  accomplish  this  during  OSD  program  reviews 

•  O  ther  AHSW6  members  take  action  within  their  Components 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 


4-6.  Assess  the  use  of  COTS  systems  engineering  tools  (modeling 
environments)  for  collaborative  architecture  development 

Lead:  OUSD(AT&L)/DS(SSE);  Support:  OASD(NII),  Components 
Products:  Revised  guidance  in  DAG;  enhanced  M&S  body  of  knowledge  for 
dissemination 
Completion  goal:  2007 


•  oyabJL  and  AP4t'3'3  already  proving  utility  in  COTS  tooJo  (market  suecees) 

•  UPlMI  nearing  finalteation,  can  help  with  CADiVJ  and  |r)/\RS  weaknesses 

•  NI3T  “Systems  Engineering  Fool  Interoperability  Plug-fiest”  underway 

•  No  inter-program  use  of  COTS  tools  for  architecture  development  thus  far 


Next  steps: 

•  Investigate  use  of  SE  tools  for  collaborative  architecture  development 

•  Propose  as  a  SBIR  topic 
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Obj.  4:  Improve  Model  &  Simulation  Use  (cont.) 


4-7.  Define  and  capture  meaningful  metrics  for  M&S  utility  in  acquisition 

Co-Leads:  OUSD(AT&L),  Dept,  of  the  Navy  Support:  OUSD(AT&L)/DS(SSE), 
Components 

Products:  Metric  definitions  in  DAG;  methods  to  capture  and  submit  data  in  DAG; 
data  from  individual  projects  in  MSIAC,  Body  of  Knowledge,  etc. 

Completion  goal:  2007 


j  One  of  the  cop  v  acquisition  j'jJ&S  projects  for  CC  rYDS  funding,  but 
didn't  make  the  out 

>  ACgis  Technologies  conducted  a  study  for  CO,  but  results  not  yet 
released 


Next  steps: 

•  Assess  adequacy  of  M&S  CO/AEgis  Technologies’  product 

•  Take  further  action  as  appropriate 
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Objective  5:  Shape  the  Workforce 


5-1 .  Define  required  M&S  competencies  for  the  acquisition  workforce 

Co-Leads:  DAU  and  OUSD(AT&L)/DS(SSE);  Support:  OUSD(P&R), 
OUSD(AT&L)/DDRE,  OUSD(C)/PA&E,  Components 
Product:  Identified  lead  FIPT;  workforce  qualification  requirements;  management 
process  &  structure 

Completion  goal:  2008 


•  “Educating  the  WIS  Workforce"  project  underway  with  Navy  and  hi &S  SC 
funding 

•  Academic  institutions  have  begun  to  leverage  this  work 

•  Participated  in  beta  version  of  SiVIU  course  “lyISS  in  Acquisition  LifecycteJ 


Next  steps: 

•  Receive  final  deliverables  from  M&S  SC-funded  project 

•  Monitor  and  assess  effectiveness  of  emerging  courses  (e.g.,  GMU) 

•  Otherwise  support  implementation  as  appropriate 
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Objective  5:  Shape  the  Workforce 


5-2.  Harvest  lessons  from  commercial  sector  activities  in  the  use  of  M&S  to 
support  product  development 

Lead:  OUSD(AT&L)/DS(SSE);  Support:  OUSD(AT&L),  Components 
Products:  Annual  update  to  best  practices  in  DAG  and  lessons  from  industry  that  should 
be  considered  by  PMs  in  planning  for  M&S 
Completion  goal:  Recurring;  initial  in  2007 


•  participating  in  conferences,  workshops,  and  (literature  review  Involving 
commercial  industry  use  of  capturing  raJavant  points 

•  Increasing  industry  adoption  of  “Simulation-Based  Design 
>  Action  complete,  but  follow-on  expansion  needed 


Next  steps: 

•  Collect  and  consolidate  findings,  feed  into  Action  5-3  BoK 

•  Submit  as  SBIR  topic 
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Objective  5:  Shape  the  Workforce 


5-3.  Assemble  and  evolve  the  M&S  Body  of  Knowledge  (information  set) 
relevant  to  acquisition 

Lead:  OUSD(AT&L);  Support:  OUSD(AT&L)/DS(SSE),  Components 

Product:  Information  base  available  to  potential  M&S  users  (e.g.,  PMs,  CMs,  OTAs); 

source  material  for  education  and  training 
Completion  goal:  Recurring;  initial  in  2006 


j  Action  completed  in  200/  no  pert  of  ongoing  education  project 

•  Severn!  -doKs  have  been  discovered 

•  M  ducat!  on  project  hoc  synthesized  a  consolidated  JdoX,  as  has  dimdummit 

•  Knowledge  is  still  being  developed  (e.g,,  best  practices) 


Next  steps: 

•  Harmonize  with  SimSummit  BoK? 

•  Establish  an  effective  configuration  management  process 

•  Make  additional  inputs  as  they  are  discovered  or  become  available 
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Obj.  5:  Shape  the  Workforce  (cont.) 


5-4.  Educate  and  train  the  workforce  to  achieve  required  M&S 
competencies 

5-4(a)  Provide  M&S  knowledge  via  an  expanded  set  of  DAU  courses, 
A*"  the  Defense  Acquisition  Guide,  and  on-line  CLMs 

Lead:  DAU;  Support:  OUSD(AT&L),  OUSD(AT&L)/DS(SSE),  Components 
Product:  Expanded  set  of  DAU  courses,  improved  M&S  guidance  in  the  Defense 
Acquisition  Guide,  on  line  Continuous  Learning  Modules;  a  better  educated 
workforce 

Completion  goal:  2009 


j  CUjJ  on  “i'jJ&o  for  Dyste/ns  Engineering"  released,  has  >3000  graduates 

•  CLM  on  “i'MS  If  or  dost  &  Evaluation"  released,  has  >‘J600  graduate© 

•  Universities  and  NIPf>  are  responding  to  “Educating  the  Workforce” 
findings  and  recommendations 

>  Mo  change  to  DAU  courses  so  far,  hut  education  project  wiii  be  a  catalyst 


Next  steps: 

•  Participate  in  prototype  GMU  course  “M&S  in  the  Acqn  Lifecycle” 

•  Implement  additional  CLMs  (Education  Project  expects  to  recommend 
-10)  as  feasible 

•  Investigate  status  of  DAG  inputs 


Obj.  5:  Shape  the  Workforce  (cont.) 


5-4.  Educate  and  train  the  workforce  to  achieve  required  M&S 
competencies 


5-4(b)  Provide  M&S  knowledge  via  conferences,  workshops,  and 
assist  visits 

Lead:  OUSD(AT&L)/DS(SSE);  Support:  OUSD(AT&L),  DAU,  Components 
Product:  Annual  outreach  program;  a  better  educated  and  trained  workforce 
Completion  goal:  Recurring;  initial  in  2006 


•  Jrmkil  Oufrooeh  Phm  approved  by  AbJbWG;  incJudoo  J VJ&b  fuforkd  for  Ab 
okifr,  DbJbC,  Nl)JA,  ond  bJbO  prooomotiono 

>  Add' I  rmuoriolo  (o.g.,  boot  pro  on  coo)  in  work 

•  Resource  constrained 


Next  steps: 

•  Advertise  and  expand  assist  visits 

•  Hold  workshops  once  recommended  practices  are  in  hand 
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Obj.  5:  Shape  the  Workforce  (cont.) 


5-5.  Improve  the  knowledge  and  expertise  available  through  the  MSIAC  to 
make  it  of  greater  utility  to  the  acquisition  community 

Lead:  OUSD(AT&L);  Support:  OUSD(AT&L)/DS(SSE),  OUSD(P&R),  OUSD(C)/PA&E, 
Components 

Product:  Plan  of  action  with  coordinated  MSIAC  CONOPS  &  staffing  requirement;  list  of 
knowledge  shortfalls  that  MSIAC  will  take  on;  success  criteria  &  process  to  bring 
MSIAC  up  to  criteria 

Completion  goal:  2008 


*  Only  preliminary  conversions  with  bibJAC  contractor  Urns  far 
>  No  pi  Ori  of  notion  by  bib  I  AC;  they  want  AblbWG  to  tell  thorn  what  to  do 


Next  steps: 

•  Develop  a  plan  of  action  to  improve  the  M&S  Information  Analysis  Center’s 
usefulness  to  the  acquisition  community 
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Backup  Material 
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AMSWG  Membership  (i  of 2) 


Organization 

SE  Forum  Principal 

Organization 

AMSWG  Member 

ODUSD  (A&T)  SSE 

Ms.  Kristin  Baldwin, 

Chair 

ODUSD  (A&T) 

SSE/DT&E 

Michael  Truelove 

Army 

Mr.  Doug  Wiltsie 

ASA(ALT) 

John  Gillis 

Navy 

Mr.  Carl  Siel 

ASN(RDA)  CHENG 
MARCORSYSCOM 

Bill  Zimmerman 

Lan -Thanh  McGough 

Air  Force 

Mr.  Terry  Jaggers 

SAF/AQR 

Maj  Carol  Beverly 

Joint  Staff  J-8 

Mr.  Rick  Westermeyer 

JTAMDO 

Jim  Gill 

PA&E 

CAIG 

DOT&E 

Dr.  Ernest  Seglie 

DOT&E 

Bob  Butterworth 

OSD  (AT&L)  DDR&E 

M&S  CO 

Jim  Anthony 

OSD  (AT&L)  AR&A 

Mr.  Phil  Rodgers 

USJFCOM 

Mr.  Steve  Derganc 

USD(P&R)(R&T)/JAEC 

R&T/JAEC 

Bob  Halayko 

OUSD(AT&L)(TRMC) 

TRMC 

George  Rumford 

DAU 

Dr.  Jim  McMichael 

DAUSE 

George  Prosnik 

MDA 

Mr.  Dennis  Mays 
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AMSWG  Membership  (2  of  2) 


Organization 

SE  Forum  Principal 

Organization 

AMSWG  Member 

NGA 

Dr.  Tom  Holzer 

NGA 

(John  Placanica-Inact) 

NASA 

Mr.  Stephen  Kapurch 

NASA/ESMD 

(Mark  Prill-Inact) 

NSA 

Mr.  Kelly  Miller 

NSA 

Craig  Holcomb 

DCMA 

Ms  Rebecca  Davies 

DCMA 

Larry  Cianciolo 

SOCOM 

Dr.  Dale  Uhler 

SOCOM 

Art  Gibson 

Nil 

Mr.  Mike  Kern 

ASD  (Nil)  Acq. 

Bill  May 

OSD  (AT&L)L&MR 

NSWC/CSS 

(Marc  Eadie-Inact) 

OSD  (AT&L)  DP&AP 

Shay  Assad 

DISA 

Mr.  Gerald  Doyle 

NSSO 

Mr.  Jay  Pamess 

NRO 

Mr.  Vernon  Grapes 

DLA 

Mr.  David  Falvey 

NDIA 

Mr.  Bob  Rassa 

NDIAM&S  Com. 

Jim  Hollenbach 

INCOSE 

Mr.  David  Walden 
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A  Decade  of  Studies  on 
M&S  Support  to  Acquisition 


1.  Final  Report  of  the  Acquisition  Task  Force  on  M&S,  1 994 

Sponsor:  DDR&E  (Dr.  Anita  Jones);  Chair:  VADM  T.  Parker,  USN  (Ret.) 

2.  Naval  Research  Advisory  Committee  Report  on  M&S,  1 994 

Sponsor:  ASN(RDA);  Chair:  Dr.  Delores  Etter 

3.  Collaborative  Virtual  Prototyping  Assessment  for  Common  Support 
Aircraft,  1995 

Sponsor:  Naval  Air  Systems  Command;  conducted  by  JHU  APL  and  NSMC 

4.  Collaborative  Virtual  Prototyping  Sector  Study,  1 996 

North  American  Technology  &  Industrial  Base  Organization;  sponsor:  NAVAIR 

5.  Application  of  M&S  to  Acquisition  of  Major  Weapon  Systems,  1996 

American  Defense  Preparedness  Association;  sponsor:  NavyAcqn.  Reform  Exec. 

6.  Effectiveness  of  M&S  in  Weapon  System  Acquisition,  1996 

Sponsor:  DTSE&E  (Dr.  Pat  Sanders);  conducted  by  SAIC  (A.  Patenaude) 

7.  Technology  for  USN  and  USMC,  Vol.  9:  M&S,  1 997 

Naval  Studies  Board,  National  Research  Council;  sponsor:  CNO 

8.  A  Road  Map  for  Simulation  Based  Acquisition,  1 998 

Joint  SBA  Task  Force  (JHU  APL  lead);  sponsor:  Acquisition  Council  of  EXCIMS 
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A  Decade  of  Studies  on 
M&S  Support  to  Acquisition 


9.  M&S  for  Analyzing  Advanced  Combat  Concepts,  1 999 

Defense  Science  Board  Task  Force  (Co-chairs:  L.  Welch,  T.  Gold) 

10.  Advanced  Engineering  Environments,  1999 

National  Research  Council;  sponsor:  NASA 

1 1.  Survey  of  M&S  in  Acquisition,  1 999  and  2002 

Sponsor:  DOT&E/LFT&E;  conducted  by  Hicks  &  Associates  (A.  Hillegas) 

12.  Test  and  Evaluation,  1 999 

Defense  Science  Board  Task  Force  (Chair:  C.  Fields) 

13.  “SIMTECH  2007”  Workshop  Report,  2000 

Military  Operations  Research  Society  (Chair:  S.  Starr) 

14.  M&S  in  Manufacturing  and  Defense  Systems  Acquisition,  2002 

National  Research  Council;  sponsor:  DMSO 

15.  M&S  Support  to  the  New  DoD  Acquisition  Process,  2004 

NDIA  Systems  Engineering  Div.  M&S  Committee;  sponsor:  PD,  USD(AT&L)DS 

16.  Missile  Defense  Phase  III  M&S,  2004 

Defense  Science  Board  Task  Force  (Chair:  W.  Schneider) 
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Assessment  of  Current  Issues/Needs 


□  Cooperative  effort  between  AMSWG  &  NDIA  M&S  Committee 

□  AMSWG  venue: 

>  Air  Force  -  Roe  (Jan  05) 

>  Army  -  Gillis,  Wallace  (Jan  05) 

>  Navy  -  Vaughn  (Feb  05) 

>  Visits  to  NAWC/AD  (ACETEF);  Army  RDECOM;  AFMC  (SIMAF,  ICE) 

□  NDIA  M&S  Committee  venue: 

>  Joint  SIAP  Systems  Engineering  Organization  (Aug  04) 

>  Future  Combat  Systems  (Sep  04) 

>  Missile  Defense  Agency  (Feb  05) 

>  Lockheed  Martin  (Feb  05) 

>  Raytheon  (Apr  05) 

>  Boeing  (Apr  05) 

>  Northrop  Grumman  (Jun  05) 

>  BAE  Systems  (Aug  05) 

□  Affirmed  many  findings  and  recommendations  from  studies  and  provided 
new  inputs  as  well 
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Top-Down  Derivation/Traceability 
to  Non-M&S  Needs 


CJCSI  3170  &  DoDD  5000.1 

Characteristics  of 
Desired  Acquisition 
Environment 


Needed  Systems 
Engineering  Capabilities 


Annotated  asAEI,  AE2,  ...  AEn 


^  Annotated  as  SE1,  SE2,  ...  SEn 


/* 


T\ 


Needed  M&S 
Capabilities 


Annotated  as  MSI,  MS2,  ...  MSn 


\  t 


% 


Ct. 


/ - 

- \ 

Gaps 

V 

_ J 

Annotated  as 
G1 ,  G2, ...  Gn 


r 

- \ 

Actions 

V 

_ J 

Annotated  as 
A1,  A2,...An 
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Desired  Acquisition  Environment: 

Key  CJSCI  31 70.01  E  Policies 

AE1 

■  Joint  conceots-centric  capabilities  identification  process  to  allow  joint 
forces  to  meet  the  full  range  of  military  operations  and  challenges... 

■  Assess  existing  anc?  proposed  capabilities  in  light  of  their  contribution 
to  future  joint  allied  and  coalition  operations.  . . .  Produce  capability 
proposals  that  consider  the  full  range  of  DOTMLPF  solutions  in  order 
to  advance  joint  warfighting  in  a  unilateral  and  multinational  context. 

■  New  solution  sets.. .crafted  to  deliver  tecliriglggically  sounds  testable , 
AE4  sustainable  and_  affordable  ijicrements  of  militarily  useful  capability. 

AE5 

■  The  FoS  and_  SoS  solutions  may  also  require  systems  delivered  by 
multiple  sponsors/materiel  developer. AE6 

■  The  process  to  identify  capability  gaps  and  potential  solutions  must  be 
supported  by  a  robust  analytical  process  AE7 

■  JCIDS  implements  a  capabilities-based  approach  that... requires  a 
AEs cgllabgratiye  process  that  utiliz^ioint  concepts  and  integrated 
ae9  ajvjTitectures  to  identify  pnontizeg_  capability  gags  and  integrated 

DOTM-P F  and_  policy  approaches  to  resolve  those  gaps 

AE11 
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Desired  Acquisition  Environment: 

DoDD  5000.1  Acquisition 

AE12 

“The  primary  objective  of  Defense  acquisition  is  to  acquire  quality  products  that 
satisfy  user  needs  with  measurable  improvements  to  rnission  capability  and 
operational  support,  in  a  timely  manner,  and  at  a  fair_  and_  reasonable  price.  ” 

AE13  AE14  AE15 

Governing  policies: 

AE1 6 

>  Flexibility,  Responsiveness  dime-phased  capabilities,  evolutionary 
acquisition),  Innovation,  Discipline,  Streamlined  Effective  Management 

>  Armaments  Cooperation;  Collaboration:  Competition;  Cost  and 
Affordability:  Cost  Realism;  Cost  Sharing;  Financial  Management; 
Independent  OTAs;  Information  Assurance;  Information  Superiority; 

ae2o  [ntegrated_  T&E;  Intelligence  Support;  Interoperability;  Knowle^gezBased 
Acquisition;  Legal  Compliance;  Performance-Based  Acquisition; 
ae23  Performance-Based  Logistics;  Products  Services  and  Technologies  [seek 
rnost  cost-effective  solution  over  the  system's  life  cycle],  Professional 
Workforce,  Program  Information  [complete,  current,  tailored];  Program 
Stability;  R&D  Protection;  Safety;  Small  Business  Participation;  Software 
Intensive  Systems;  Streamlined  Organizations;  Systems  Engineering; 
AE26Technglogy  Deyelgprnent  and_  Transition;  Total  Systerns  Approach  ae27 


>  Oct  04  policy  memo:  Technical  reviews  ...  shall  be  event-driven  ae28 
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Necessary  Systems  Engineering  Capabilities 

(which  M&S  can  affect;  derived  from  Desired  Acquisition  Environment) 

sei.  Early,  continuing  systems  engineering  from  an  SoS/FoS  capabilities 
perspective;  seamless  transition  from  JCIDS  to  acquisition 

(AE1-3, 5, 9-11, 16, 20, 21 ,25,27) 

SE2.  Lifecycle-wide  exploration  of  the  maximum  available  trade  space, 
including  time-phased  requirements  and  technology  insertion 

(AE1  -5,7,1 0,1 1 ,1 3,1 6,1 9,23-27) 

SE3.  Collaboration  among  all  stake  holders  (multiple  gov’t  and  contractor 
organizations)  for  key  enterprise-level  SE  decisions  <ae6-8,  10,18,22, 25, 27) 

SE4.  Rapid  assessment  of  concept/design  alternatives  (AE2, 4, 7, 10, 14,16,19,25, 28) 

SE5.  Comprehensive,  accurate,  event-based  assessment  of  technical 
baselines;  avoidance  of  costly  fixes  for  problems  discovered  late 

(AE2-4,7,9,1 0,1 2-1 7,1 9,20,22,24-26,28) 

SE6.  Focused,  effective  &  efficient  testing;  including  at  the  capability  level 

(AE1 ,2, 4, 5, 9-1 1 ,13,15,1 9-22,25) 

SE7.  Appropriate  reuse  of  all  resources  -  information,  software  tools, 
expertise,  facilities,  ranges,  etc.  -  across  programs  &  organizations 

(AE4, 14, 15, 19, 24, 25) 
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Needed  M&S  Capabilities  (iof2) 

(derived  from  Needed  Systems  Engineering  Capabilities) 

MSI.  Model-based  systems  engineering/design  (sei,2,4,5) 

(Emerging  concept  under  INCOSE,  OMG,  etc.;  growing  suite  of  COTS  tools) 

>  Modeling  environments  to  analyze  requirements,  develop  system  and 
software  architectures,  and  perform  detailed  design  (e.g.  CAD,  S/W) 


MS2.  M&S-enabled  collaborative  engineering  environments  (sei,2,3,4,s,6) 

>  Interoperable  M&S,  data 
management,  &  manufacturing 

■  M&S  as  a  communication  means 

>  Full  range  of  M&S  assessments 

■  Models,  simulations,  and  distributed 
live-virtual-constructive  simulation 


Capability 
System  — 1 

Engineering  j 


Program 

System 

Engineering 


A 


Capability/ 
Verif.  / 


TrchT 

Vert. 


Co  ordinal  e/sup  port  Development  and  Engineering  Changes 


^Capability 

Evaluation 


System 

T&E 


federations,  with  option  to  immerse  warfighters 
>  Traceability  for  coherence  and  decision  analysis 


MS3.  Model-Test-Fix-Model  process  across  the  life-cycle  (se4,s,6) 

>  Better  test  planning,  more  effective  tests 

>  Increased  M&S  validity;  credible  surrogates;  reuse  savings 
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Needed  M&S  Capabilities  (2  of  2) 

MS4.  M&S  knowledge  to  formulate  an  effective  acquisition  strategy  (se2,3,4,5,7) 

>  Ready  access  to  M&S  expertise  and  information  about  capabilities 
and  gaps,  reusable  resources,  lessons-learned,  etc. 

MS5.  Disciplined  M&S  planning  &  employment  (SE2, 4, 5,7) 

>  Rigorous  analysis  of  M&S  requirements,  alternatives,  best  course 

>  Efficient  configuration/initialization,  execution  and  post-run  analysis 

>  Avoid  inappropriate  use;  maximize  cost-effective  reuse  across  lifecycle 

MS6.  Efficient  development/evolution  of  credible  M&S  tools  jse2,3,s,7) 

>  A  systems  engineering  approach  with  appropriate  V&V 

MS7.  Access  to  authoritative,  understandable  data  needed  for  M&S 
representations  (se2,3,4,s,7) 

>  Reducing  a  major  time  and  cost  burden  that  inhibits  M&S  use 

MS8.  Inspection  of  M&S  used  to  inform  acquisition  decisions  (se2,s,7) 

>  Examine  capabilities  and  limitations  (W&A)  of  M&S 

>  During  lead-up  to  program/technical  reviews,  OTRRs,  DABs,  etc. 
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Gaps 

1.  Management 

G1.  Robust  but  confused  landscape  of  M&S  activities;  no  clearly 
designated  leadership  or  effective  coordinating  mechanism  <msi-8) 

>  Current  EXCIMS  ineffective;  little  coordination  for  capabilities/SoS/FoS 

G2.  Inadequate  constancy  of  purpose  because  time  to  fix  problems  » tour 
length;  “DoD  has  an  attention  deficit  disorder”  (MS2-7) 

G3.  Gov’t  acquisition  guidelines  don’t  promote  M&S  use  or  reuse  <msi-6) 

G4.  No  DoD  requirement  for  formal  M&S  planning  to  support  acquisition 
(other  than  T&E)  (msi-5) 

G5.  No  contractual  guidelines  regarding  M&S  and  the  data  it  needs  (msi-8) 
G6.  Gov’t  typically  doesn’t  give  contractors  meaningful  M&S  guidance 

(MSI  ,2,6,8) 

G7.  Most  DoD  M&S  takes  a  project,  vice  an  enterprise,  approach  <ms2,3,6,7) 
G8.  No  consensus  on  value  of  integrated  architectures,  nor  responsibility 

for  (MSI  ,2) 

G9.  Managing  distributed  collaboration  is  very  hard  (msi-8) 

G10.  Public  law  precludes  OT  based  solely  on  M&S,  but  no  clear  guidance 
on  use  for  SoS/FoS  T&E  <ms2,3,5,6,8) 
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Gaps 


2.  Architecture/standards/technical  framework 

G11.  No  standard  modeling  notation  (like  UML  v2.0)  for  capturing  full  range 
of  information  critical  to  system  engineering  (e.g.,  structure,  behavior, 
requirements  hierarchy/traceability,  test  cases,  verification  results)  (msi,2,6,7) 

G12.  No  standard  for  interchanging  systems  engineering  information  (same 
examples  as  above)  (msi,2,6,7) 

G13.  No  conceptual  framework  (like  Open  System  Interconnect  protocol  stack) 
for  data  interchange  (msi,2,3,6,7) 

G14.  Lack  of  agreement  on  a  common  distributed  simulation  standard 
increases  complexity  and  cost,  limits  simulation  interoperability  (MS2,5,6) 

G15.  DoDAF  vl  .0  is  difficult  to  use  for  architecting  due  to  lack  of  data- 
centricity  and  executability;  some  products  of  marginal  value  (msi,2,6,7) 

G16.  Use  of  DoD-unique  standards  limits  their  user  base,  quality,  COTS  tool 
support,  and  opportunities  for  reuse  (msi,2,5,6) 


70 


Gaps 

3.  Model/simulation  capabilities  &  use 

G17.  Many  M&S  tool  gaps  and  deficiencies  (msi,2, 3,5,7) 

>  What’s  modeled  (e.g.,  urban  warfare,  comm  networks,  threats,  system  sustainment) 

>  Fidelity,  granularity,  interoperability 

>  Only  limited  consensus  on  common  models  to  be  used  across  a  domain 

G18.  No  good  way  to  develop  and  maintain  widely-needed  M&S  tools  that  cut 
across  programs  (mss,6) 

>  Not  incorporating  mods  by  other  organizations  into  “street  version,”  etc. 

G19.  M&S  developers,  not  M&S  users,  tend  to  drive  M&S  development  (MS6) 

G20.  In  general,  architecture  development  (modeling)  is  lagging,  not 

collaborative,  and  not  exploiting  COTS  SE  tools  (modeling  environments) 

(MSI, 2) 

G21.  No  readily-available  distributed  M&S  infrastructure  (e.g.,  JDEP)  (MS2,5) 

G22.  Hard  to  get  security  certification  for  multi-organization  (company/ 
Service)  distributed  simulation  (MS2,3,5,6) 

G23.  Hard  to  get  approval  and  security  certification  for  M&S  involving 
multiple  compartmented  programs  (SAPs)  (MS2,3,5,6,7) 
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Gaps 


4.  TrustworthinessA/V&A 

G24.  Post-development  model  validation  expensive  and  slow  (MS2,3,5,8) 
G25.  VV&A  often  weak  or  non-existent;  documentation  inconsistent 

(MS2,3,5,8) 

>  Plans  to  use  M&S  to  avoid  testing  costs  often  rejected  due  to  poor/no 
validation 

G26.  VV&A  usually  not  enforced  and  also  not  examined  during  program 

reviews  (MS2,3,5,6,8) 

G27.  Models  and  sims  often  not  updated  to  reflect  empirical  evidence 
(e.g.,  test  results)  (ms2,3,5,8) 
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Gaps 

5.  Sharinq/reuse  and  protection  of  tools  &  information 

G28.  Little  reuse;  only  7%  of  models  &  sims  used  on  >1  program  (MS2,5,6) 

G29.  Concurrent  engineering  requires  an  integrated  process,  data  sharing 
and  a  coherent  tool  set,  but  <20%  of  programs  have  such  a  collaborative 
environment  (MS2,7) 

G30.  Hard  to  discover  reusable  resources  (software,  info,  services)  <ms2,4,5,7) 

>  M&S  repositories  are  not  integrated,  lack  an  effective  search 
capability,  and  are  mostly  empty 

>  MSIAC  knowledge/expertise  is  lacking 

G31.  Insufficient  info  (metadata)  to  evaluate  data/reuse  candidates  <ms2,4,5,7) 

G32.  Hard  to  obtain  reusable  resources  <ms2,4,5,7) 

>  Industry  to  gov’t:  To  protect  proprietary  info  &  competitive  advantage 

>  Gov’t  to  industry:  Contractual  liabilities  associated  with  GFE/GFI 

>  Gov’t  to  gov’t:  Concerns  about  misuse;  cost  to  deliver  and  guide 

G33.  No  incentives  to  encourage  reuse  (MS2,3,5,6) 

>  Negative  incentives  include  cost  to  make  reusable,  workload 
assisting  users,  vulnerability  to  criticism 

[plus  approval  and  security  certification  gaps  22  &  23  listed  under  M&S  use] 
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Gaps 


6.  Resea  rch/S&T/tech  base 

G34.  Conceptual  foundation  of  M&S  weak  (mss,6) 

>  E.g.,  theoretical  understanding  of  modern  warfare,  human 
behavior,  relating  M&S  at  different  granularities,  dealing  with 
uncertainty,  agent-based  modeling  and  generative  analysis 

G35.  Little  acquisition  community  input  to  DoD  S&T  management 
regarding  needed  M&S-related  research  (ms2,5,6) 

7.  Business  model,  metrics  &  ROI.  funding  and  incentives 

G36.  No  business  model  for  how  M&S  capabilities  should  be  developed, 
used  and  maintained  (msi-8) 

G37.  Metrics  are  critical  to  keep  interest  and  funding  up,  but  metrics 
regarding  M&S  use  and  cost-effectiveness  are  inadequate  (msi-8) 

>  M&S  funding  difficult  to  identify;  most  embedded  within  other  PEs 

G38.  Too  little  funding  (MS2-7) 
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Gaps 


8.  Workforce  Shaping 

G39.  Body  of  knowledge  for  M&S  support  to  acquisition  is  deficient,  not 
managed  (msi,2,4-6,8) 

G40.  Acqn  community  managers  and  staffs  mostly  uninformed  about 
M&S  capabilities  and  limitations  (msi-8) 

>  Weak  acquisition  personnel  understanding  of  commercial  M&S 
activities  (“We  don’t  get  out  enough”) 

>  Not  enough  M&S  experts  (no  career  path  [except  Army],  no 
formal  education  or  training) 

G41.  M&S  developers  lack  understanding  of  modeling  best  practices, 
abstraction  techniques,  context  dependencies,  etc.  (MS3,6) 

G42.  M&S  users  often  not  adequately  trained  (msi,2,4,5,8) 

G43.  Insufficient  M&S  education  options  (MS2,4,5,6,8) 
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Virtual  Battlespace  Center 

Advanced  M&S  to  Inform 
OSD  Acquisition  Decisions 


NDIA  Systems  Engineering  Conference 

San  Diego,  California 
20-23  October  2008 


James  W.  Hollenbach 
ODUSD(A&T)/SSE/DTE  Support 

Simulation  Strategies,  Inc. 
jimh@simstrat.  com,  727.824.0931 


Outline 


Observations 

Virtual  Battlespace  Center  Concept 
Modeling  and  Simulation  in  the  VBC 
Protecting  Business-Sensitive  Information 
Organization  Options 
Influence  on  Other  M&S 
DSB  Task  Force  Study 


Observations  <iof3) 


1.  Acquiring  capabilities  is  a  huge  challenge 

•  Expanded  trade  space  =  dramatic  increase  in  complexity 

-  Many  more  entities,  variables,  interactions,  etc. 

-  Development  enterprises  become  vast,  distributed 

-  Many  more  stakeholders,  with  much  at  stake 

•  Need  systems  engineering  above  individual  system  level 

-  Complexity  precludes  intuitive 
design  and  analysis 

-  Program  to  program  negotiation: 


Need  to  assess  capabilities, 
not  just  individual  systems 


impractical 


to  assess  capabilities,  ~~ 

st  individual  systems  Program 


Coordinate  Development  and  Engineering  Changes 


■  Vei  SoS 


/* 


Evaluation 


-  Many  more  forces  &  systems,  Engineering 


-  Scarcity  of  equipment  constrains 
lab  integration  &  live  tests 


bigger  battlespace,  more  events 


System 


System 

T&E 


—  Range  size,  security  needs,  and 
safety  also  limit  live  testing 
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Observations  (2  of  3) 

2.  M&S  can  improve  the  design,  integration,  test,  and  assessment  of 
capabilities  and  systems  of  systems 

•  Earlier,  more  accurate  designs  and  assessments  of  designs,  lowering  risk 

•  Accurately  track  complex  relationships  and  micro-level  interactions; 
present  macro-level  measures  of  merit  to  decision  makers 

•  Defendable  analytical  underpinning  for  decisions 

•  Several  SoS  efforts  (e.g.,  JSSEO,  FCS)  provide  glimpses  of  M&S  benefits 

3.  DoD  prime  contractors  have  built  joint  battlespace  simulations  to 
help  develop  new  warfighting  capability/system  concepts  and  to 
collaborate  with  their  government  customers  &  industry  partners 

•  But  these  virtual  battlespaces  are  neither  authoritative  nor  coherent 

-  Representations  of  blue  systems  they  don’t  build  aren’t  authoritative 

-  They  have  different  conceptual  models  of  the  battlespace  and  standards 

•  Thus  many  of  their  intended  benefits  are  negated 

-  Inaccuracies  can  lead  to  bad  business  decisions 

-  Government  customers  question  their  credibility 

-  Collaboration  with  partners  is  hampered  by  incoherent  representations 
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Observations  (3of3) 


4.  The  Services  also  have  various  SoS  simulation  efforts  to  support 
their  system  development  activities,  such  as: 

•  Army’s  Cross-Command  Collaboration  Effort  (3CE)  and  Modeling  Architecture 
for  Technology,  Research  and  Experimentation  (MATREX)  Distributed  Virtual 
Laboratory  (DVL) 

•  Air  Force  Integrated  Collaborative  Environment  (AF-ICE) 

5.  OSD  has  no  equivalent  virtual  battlespace... 

•  to  provide  independent  assessments  of  system  concepts  and  designs 

•  to  plan  and  evaluate  how  individual  systems  function  in  a  SoS 

•  to  assess  capabilities  as  proposed  and  as  evolutionary  changes  to  a  SoS 
occur 

6.  Hence  corporate  decisions  are  not  as  informed  as  they  could  be 
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Virtual  Battlespace  Center  Concept 


•  Primary  mission:  Support  OSD's  corporate-level  acquisition 
responsibilities  with  advanced  M&S 

-  A  persistent  environment  in  which  all  DoD-level  capability/  system  of 
systems  (SoS)  design  and  analysis  is  conducted 

-  A  means  to  refine  concepts  and  define  requirements  for  both 
capabilities  and  individual  systems 

-  An  objective  view  of  how  systems  interoperate  and  perform 

•  Secondary  mission:  Support  analysis  of  DoD  investment  decisions 
and  operational  plans 

•  VBC  will  have  the  most  credible  representations  of  every  system, 
force,  and  activity  in  the  battlespace 

•  To  do  so,  VBC  must  provide  security  for  business-sensitive 
information 
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VBC  Modeling  and  Simulation 

*  A  wide  range  of  M&S  will  be  used  in  the  VBC 

-  Architecture  modeling 

•  Design  SoS  topology,  allocate  functions,  check  completeness  &  efficiency 

•  Using  SE  tools  like  System  Architect,  Core,  Cradle 

-  Concept  assessment  modeling 

•  Comprehensive  view  of  the  entire  trade-space  to  assess  design  decision 
impacts  on  key  performance  parameters  (KPPs) 

•  E.g.,  Georgia  Tech’s  Collaborative  Visualization  Environment  (CoVE) 

-  Distributed  simulation 

•  Any  mix  of  live,  virtual,  and  constructive  (LVC)  simulation;  much  constructive 
simulation  will  be  other  than  real-time 

•  Multiple  standard  federations  will  provide  an  80%  solution 

-  Recursive  levels  of  granularity  for  models,  simulations,  and  federations 

•  Based  on  hierarchically-integrated  conceptual  models  (common  in 
engineering,  but  thus  far  quite  uncommon  in  M&S) 

-  Other  M&S  TBD 
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Why  Won’t  JMETC  Suffice? 

•  Joint  Mission  Environment  Test  Capability  provides  distributed 
simulation  for  operational  testing  of  systems 

-  Managed  by  the  Testing  in  a  Joint  Environment  Senior  Steering  Group 

•  Provides  a  sub-set  of  M&S  capabilities  needed  by  VBC 

-  Doesn’t  include  architecture  modeling  or  concept  assessment  modeling 

-  Distributed  simulation  is  limited  to  real-time  (TENA-based) 

-  Is  focused  on  a  single  level  of  granularity  (platform  level) 

-  Won’t  adequately  protect  business-sensitive  information 
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Need  for  Trustworthy  Representations 

•  Corporate-level  decisions  arising  from  VBC  analyses  will  determine: 

-  what  individual  systems  are  procured  or  modernized; 

-  the  functional  capabilities  &  interfaces  each  system  must  have; 

-  the  standards  to  which  those  systems  must  conform; 

-  the  schedule  on  which  they  must  be  developed  or  evolved;  and 

-  indirectly,  the  funding  allocated  to  each 

•  The  risk  of  an  erroneous  representation  leading  to  an  incorrect 
decision  must  be  minimized 

-  Decisions  will  be  challenged  unless  the  VBC  representations  and 
associated  analyses  are  above  reproach 

•  VBC  must  have  credible,  trustworthy  representations 

-  System  data  &  algorithms  must  be  traceable  back  to  the  most  credible 
sources 

•  Program  offices  &  their  contractors  should  be  tasked  to  supply 
validated  representations  of  their  systems 

-  Requirements  for  these  must  first  be  carefully  and  unambiguously 
defined  by  the  VBC 
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Protecting  Business-Sensitive  Information 

•  To  get  trustworthy  representations,  business  sensitive  information 
(e.g.,  intellectual  property,  programmatic  info)  must  be  protected 

-  Contractors  fear  compromising  IP,  undermining  business  opportunities 

-  Program  offices  are  concerned  their  program  or  reputation  will  be  harmed 

-  VBC  must  assure  representation  resource  owners  that  misuse  or 
compromise  will  not  occur 

•  Distributed  simulation  technology  provides  some  protection  of 
sensitive  information  via  encapsulation,  but  it  can  still  be 
compromised  by  repeated  observation  of  a  system’s  behaviors 

-  For  instance,  multi-sensor  integration  logic  can  be  inferred  from 
responses  to  various  patterns  of  sensor  inputs 

•  To  prevent  industrial  espionage,  VBC  will  have  to  tightly  control 
access  to  its  M&S  representations 

-  VBC  personnel  will  operate  the  models  and  simulations,  or 

-  Other  simulation  owners  participating  in  a  distributed  simulation  will  do 
so  under  non-disclosure  agreements,  with  tightly  limited  data  collection 
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Organization  Options 


Capability/SoS  management  may  be  instituted  under  an  evolved 
portfolio  management  concept  or  other  organizational  structure 

-  VBC  would  support  whatever  organization  emerges 

Candidate  organizations  to  run  the  VBC  include: 

-  Existing  OUSD(AT&L)  office 

-  Defense  agency  or  field  activity,  either  existing  or  new 

-  System  command  of  a  DoD  Component  (objectivity  concerns) 

-  FFRDC  or  UARC 

-  Contractor  recused  from  any  other  system  acquisition  work 

-  Fire-walled  division  of  a  contractor  (objectivity  concerns) 

Selection  will  require  further  study 


Impact  on  Other  M&S 


•  System  assessment  in  the  VBC  would  be  a  capstone  event 

-  The  system’s  performance  and  contribution  to  a  desired  DoD  functional 
capability  will  be  evaluated  and  its  fate  decided 

•  System  owners  will  want  their  own  virtual  battlespace  to  be  as  close 
as  possible  to  the  VBC’s,  so  standards  used  in  the  VBC  will  foster 
alignment  by  the  rest  of  acquisition  M&S 

-  Architectures,  battlespace  conceptual  models,  &  FOMs  can  be  matched 

-  Government-owned,  non-IP  data  used  in  VBC  (e.g.,  scenarios,  threats, 
natural  environment)  can  be  shared  under  CRADAs 

-  “One-off  versions”  of  owner-provided  representations  could  be  shared 
using  abstraction  means  (e.g.,  neural  nets,  response  surface  equations) 

-  VV&A  practices  to  ensure  the  trustworthiness  of  VBC  representations  will 
foster  more  diligent  W&A  in  other  virtual  battlespaces 

•  Interoperability,  reuse,  and  rapid,  cost-effective  composition  of 
distributed  simulation  federations  will  all  be  enhanced 

-  As  a  side  benefit,  we’ll  achieve  effective  DoD  M&S  governance 

-  Benefiting  DoD’s  mission,  our  warfighters,  and  the  nation 
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Next  Step:  DSB  Task  Force  Study 


•  A  Defense  Science  Board  Task  Force  is  about  to  be  convened  to... 

-  examine  and  refine  the  Virtual  Battlespace  Center  concept 

-  consider  capability  management  approaches 

-  develop  a  VBC  concept  of  operations 

-  identify  and  prioritize  candidate  M&S  capabilities 

-  recommend  an  organization  to  manage  the  VBC 

-  verify  or  refute  the  VBC  benefits  asserted  here 

•  Dr.  Anita  Jones  and  Dr.  Ron  Sega,  both  former  DDR&E’s,  will 
co-chair 

•  If  the  DSB  TF  makes  a  positive  recommendation,  this  will  set  the 
stage  for  a  VBC  initiative  under  the  next  administration 
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SSE 


Systems  and  Software  Engineering 


Integration  of  Systems  and  Software  Engineering: 

Implications  from  Standards  and  Models  Applied 

to  DoD  Acquisition  Programs 

NDIA  11th  Annual 
Systems  Engineering  Conference 
San  Diego,  Ca. 

October  23,  2008 

Donald  J.  Gantzer 
Lisa  Reuss 

Systems  Engineering  Support  Office 
supporting 

ODUSD(A&T)  Systems  and  Software  Engineering  (SSE) 
Assessments  and  Support 
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Agenda 


Systems  and  Software  Engineering 


>  Introduction 

>  Overview  comparison  of  Systems  Engineering  (SE)  process 

standards  and  models 

>  Some  observations  from  review  of  SE  Plans  (SEPs) 

>  Some  findings  from  Program  Support  Reviews  (PSRs) 

>  NDIA  Summary  of  SE  and  Software  issues  in  DoD 

>  Summary  implications  of  SE  processes  in  DoD  Acquisition 
Programs 


Disclaimer:  The  views  and  opinions  presented  here  are  the  authors’ 
and  do  not  necessarily  represent  DoD  views. 
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Introduction 


Systems  and  Software  Engineering 


>  DoD’s  Defense  Acquisition  Guidebook  (DAG)  has  applied  SE 
standards  in  developing  its  SE  chapter  4,  tailored  to  DoD  acquisition 
policies  and  guidance 

>  ISO/IEC  15288  -  System  life  cycle  processes  was  recently 
updated  and  is  in  concert  with  an  update  to  ISO  12207  -  Software 
life  cycle  processes  (further  “Harmonization”  is  ongoing) 

>  A  DAG  update  is  imminent  with  changes  due  to  new  DoD 
acquisition  policies  [e.g.,  DoDI  5000.2)  that... 

•  Emphasizing  enhanced  (i.e.,  early]  Systems  Engineering  (SE) 

•  Moving  Milestone  B  acquisition  decision  point  to  post  Preliminary 
Design  Review  [PDR] 

•  Changes  to  SE  processes  per  ISO/IEC  15288  revision 


Note:  Acronym  definitions  are  in  backup  slides 
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A  Generic  SE  Process 


Systems  and  Software  Engineering 


Note:  Was  applied  to  Air  Force  IT/CSE  SE  Case  Studies;  http://www.afit.edu/cse/ 


Sources:  DoD  Mil  Std  499 A/B  and  the  Defense  Acquisition  University  [DAU]  SE  Fundamentals,  2001 
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ystem 
SO/I  EC 


Life  Cycle  Processes 
15288:  2008* 


*Changes  highlighted  in  red 


Agreement 

Processes 


Acquisition  Process 

Supply  Process 

Project- 

Enabling* 

Processes 


Life  Cycle  Model 
Management  Process 


Infrastructure 
Management  Process 


Project  Portfolio 
Management  Process 


Human  Resource 

Management  Process 


Quality  Management 
Process 


Source:  htto://www.  iso,  org/iso/iso  cataloguejitm 
(search  ‘15288’). 


Project  Processes 


Project  Planning 
Process 


Project  Assessment 
and  Control  Process 


Decision  Management 
Process 


Risk  Management 
Process 


Configuration 
Management  Process 


Information  Management 
Process 


Measurement  Process 


Technical 

Processes 

Stakeholder 
Requirements 
Definition  Process 

Requirements  Analysis 
Process 


Architectural  Design 
Process 


Implementation 

Process 


Integration  Process 


Verification  Process 


Transition  Process 


Validation  Process 


Operation  Process 


Maintenance  Process 


Disposal  Process 


Systems  and  Software  Engineering 


Note:  Also  adopted  by  IEEE,  | _ 

http://standards.  ieee.  org/announcements/pr_  15288.  html 
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Simplified  Application  of  the  SE  ‘V’ 
approach 


Systems  and  Software  Engineering 


[Note:  new  DAG 
revision  release 
imminent;  tailored 
approach  adapting 
ISO  15288:  2008  ] 
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[Source:  DoD  DAU/DAG;  Chapter  4  on  SE;  11/04] 
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SE  Standards  Example  Mapping 
-  Management  (*see  also  other  DAG  chapters) 


Systems  and  Software  Engineering 


CMMI®-ACQ  [level  3] 

DAG/SE  (#4)  | 

Project  Planning 

Planning 

Planning  and 
integrating  the 
technical/SE  effort 

Project  Planning;  Integrated 
Project  Management  (Mgt.). 
Acquisition  Technical  Mgt. 

Technical  Planning 
(*see  #11  Program 
Management) 

Project 

Assessment  and 
Control 

Assessment; 

Control 

Control; 

Technical  reviews 

Project  Monitoring  and 

Control 

Technical 

Assessment; 

Interface  Mngt.* 

Measurement 

Systems 

Analysis 

Control 

Measurement  and  Analysis 

Decision  Analysis* 

Decision 

Management 

Systems 

Analysis 

Systems  analysis 

Decision  Analysis  and 
Resolution 

Decision  Analysis* 

Risk  Management 

Systems 

Analysis 

Systems  analysis 

Risk  Management 

Risk  Management* 

Configuration 

Management 

(CM) 

Control 

CM;  integrated 
repository;  System 
breakdown  structure 

CM; 

Requirements  Management 

CM; 

Requirements 

Management 

Information 

Management 

Integrated  data 
package 

Project  Planning; 

Measurement  and  Analysis 

Technical  Data 
Management 

Acquisition  and 
Supply 

Acquisition 
and  Supply 

Development 

strategies 

Agreement  Mgt. 

Acquisition  Technical  Mgt. 

(*see  other  DAG 
chapters) 

Project  -Enabling 
processes 

Environment 

and 

Enterprise 

Support 

Product  and  process 
improvement; 

Quality  Mgt. 

Organizational  Process  set; 
Process  &  Product  Quality 
Assurance; 

Organization  Training 

(*see  other  DAG 
chapters  ) 

SE  Standard  Example  Mapping  - 
Technical  (*see  also  other  DAG  chapters) 


Systems  and  Software  Engineering 


DAG/SE  (#4)* 

Stakeholder 

Requirements 

Definition 

Requirements 

Definition 

Requirements  analysis 

Acquisition  Requirements 
Development; 

Solicitation  &  Supplier 
Agreement  Development 

Stakeholder 

Requirements 

Definition* 

Requirements 

Analysis 

Systems 

Analysis 

Requirements  & 

Functional  analysis; 
Systems  Analysis; 

Modeling 

Acquisition  Requirements 
Development 

Requirements 

Analysis* 

Architectural 

Design 

Solution 

Definition 

Functional  analysis; 
Synthesis;  Modeling, 
Specifications/drawings 

Technical  Solution 

Architecture  Design 

Implementation 

Implementation 

Prototyping;  fabrication, 
assembly,  production 

Integrated  Project 
Management 

Implementation 

Integration 

Integrated  data  package; 
Integration 

Integrated 

Project  Mngt. 

Integration 

Verification 

System 

Verification 

Functional  &  Design 
verification;  Technical 
reviews;  Test 

Acquisition  Verification 

Verification 

(*see  #9  -  Integrated 

Test  &  Evaluation) 

Validation 

Requirements  & 
End  Products 
Validation 

Requirements  validation; 
Test 

Acquisition  Validation 

Validation 

(*see  #9  -  Integrated 
Test  &  Evaluation) 

Transition 

Transition  to  Use 

Transition 

Operation; 

Maintenance; 

Disposal 

Support  stage 

*see  other  DAG  (e.g., 

#5  Life  Cycle  Logistics) 

Systems  and  Software  Engineering 


Summary  of  SE  Standards 


>  ISO/IEC  15288  is  becoming  the  leading  SE  ‘Standard’ 

•  ISO/IEC  12207  [and  others  -  e.g.,  15939  re  Measurement 
process]  working  for  ‘harmony’ 

•  IEEE  ‘adopts’  it  with  tailoring  guidance;  expect  revision  of  IEEE 
1220  [also  for  EIA-632] 

•  INCOSE  adopts/tailors  it  with  much  more  detail 

•  DoD’s  DAU  DAG  applies  it  with  acquisition-oriented  tailoring 

•  It  is  a  standard  and  so  is  a  very  high  level  ‘What’  is  best  practice 

>  “reality  is  in  the  details” 

-  the  DAG,  CMMI,  and  INCOSE  all  provide  more  details  on  what  & 
how 

>  Next  -  overview  of  DUSD(A&T)  SSE  (and  NDIA)  observations  and 
finding  regarding  SEP  reviews,  PSRs  analysis,  workshop  findings 
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USD(AT&L)  Systems  Engineering  Plan 
(SEP)  Policy* 


Systems  and  Software  Engineering 


“Provide  a  context  within  which  I  can  make 
decisions  about  individual  programs.” 
“Achieve  credibility  and  effectiveness  in  the 
acquisition  and  logistics  support  processes.” 
“Help  drive  good  systems  engineering 
practices  back  into  the  way  we  do  business.” 


Programs  shall  develop  a  Systems  Engineering 
Plan  (SEP)  for  Milestone  Decision  Authority 
(MDA)  approval  in  conjunction  with  each 
Milestone  review,  and  integrated  with  the 
Acquisition  Strategy.  This  plan  shall  describe  the 
program ’s  overall  technical  approach,  including 
processes,  resources,  metrics,  and  applicable 
performance  incentives.  It  shall  also  detail  the 
timing,  conduct,  and  success  criteria  of  technical 


reviews. 


JAi 


THE  UNOER  SECRETARY  OF  DEFENSE 


MEMORANDUM  FOR:  SEE  DIS1 R1BITION 
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Note:  colors  are  authors 


Tull  policy  can  be  found  at 

http://www.acq.osd.mil/sse/policy.html 


SEP  Purpose 


Systems  and  Software  Engineering 


>  The  SEP... 

•  Is  the  artifact  of  a  program’s  technical  planning 
activities  usually  led  by  a  SE  Working  Integrated 
Product  Team  [SE  WIPT] 

-  Captures  government  processes  and  planning 

-  Establishes  roles,  responsibilities,  and  authorities  of  both 
government  personnel  and  contractors  within  government 
processes 

•  Covers  the  life  cycle  from  concept,  acquisition,  etc., 
through  sustainment  of  the  system/program 

•  Is  the  Program  Manager’s  technical  management  tool 


Application  of  a  quality  technical  planning  process,  by  trained  and 

experienced  staff,  leads  to  a  good  SEP 


Technical  Planning  Focus  Areas  in  SEPs 

[there  are  variations  per  Milestone  /  Phase] 


Systems  and  Software  Engineering 


>  Program  Requirements 

•  Capabilities,  CONOPS,  and  key  performance  parameters/attributes 

•  Statutory,  Regulatory,  Certification  requirements 

•  Technology  development,  design  considerations 

•  Data  to  monitor  &  compare  to  assumptions 


>  Technical  Staffing  and  Organizational  Planning 

•  Lead/Chief  SE  &  functional  Leads 

•  IPT  Organization/Structure,  staffing  &  skills,  coordination 

•  Integration  with  contractors  &  external  organizations 


>  Technical  Baseline  Management 

•  Technology  maturity  &  risk 

•  Technical  Baseline  management  responsibility  &  control 

•  Requirements  traceability,  verification  &  validation 

•  Specifications  &  Work  breakdown  Schedule  (WBS) 

>  Technical  Review  Planning  [Event  driven] 

•  Technical  review  management  (who  chair,  determines  readiness  &  closure) 

•  Entry  and  exit  criteria 

•  Stakeholder  participation 

•  Peer  participation  [e.g.,  independent  Subject  Matter  Experts  [SMEs]] 

>  Integration  with  Overall  Program  Management 

•  Linkage  to  other  program  plans  (e.g.,  Acquisition  Strategy,  Integrated  Master  Plan  & 
Schedule,  Test  &  Evaluation  Management  Plan  (TEMP),  production,  sustainment/logistics 
plans  or  strategies,  etc.) 

•  Risk  Management  Plan 

•  Contracting  Considerations  (e.g.,  SE  incentives)  Source:  ODUSD(A&T)SSE  Systems  Engineering 

12  Plan  (SEP})  Preparation  Guide;  2008 


Critical  &  Substantive  Comments  per  SEP 


Systems  and  Software  Engineering 


Comments  per  SEP  by  Focus  Area 
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N=58 


Have  seen  little  significant  improvement  in  trend 
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SEP  References  to  SE  Processes 


Systems  and  Software  Engineering 


SEP  references  to  SE  process  standards  &  models 
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‘other  CMM  =  CMMI-AM,  SW-CMM 


Note:  N  =40;  sum  >100%  due  to  several  listed  in  many  SEPs 
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Systems  and  Software  Engineering 


SEP  Observations  on  SE  Practices 


>  CMMs  clearly  dominate;  but  simplified  ‘”V”  or  499B  are  still  applied 

-  some  use  the  CMMI-Acquisition  Module  (CMMI-AM); 

-  CMMI  for  Acquisition  (CMMI-ACQ)  is  too  new  to  see  any  application 

>  Some  programs  list  many  -  but  not  clear  which,  if  any,  actually  are 
applied 

>  ~  20%  not  referencing  any  particular  SE  standard/model 

>  Practically  no  information  on  what/how  standards/models  were 
tailored 

>  Some  programs  are  referencing  (adopting?)  the  Prime/Integration 
contractor’s  SE  set  of  processes 

>  SEPs  usually  only  show  or  discuss  in  detail:  Requirements 
Management,  Configuration  Management,  Risk  Management,  & 
Technical  Review  approaches  (T&E  is  addressed  in  the  TEMP) 

Need  to  see  more  details  on  tailored  integrated  SE  approach 
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SEP  Review  Summary  Observations 

(~  100  SEPs  reviewed  across  life  cycle  phases) 


Systems  and  Software  Engineering 


>  Lack  complete  requirements  ([e.g.,  regulatory,  statutory,  certifications)  sources 

>  Unclear  understanding  of  interfaces/coordination  with  other  programs/systems 
[i.e.,  System  of  Systems,  Family  of  Systems  (SoS/FoS)) 

>  Inadequate  linking  of  Key  Performance  Parameters  (KPPs),  Attributes,  Technical 
Performance  Measurements  (TPMs) 

>  Vague  on  design  considerations  and  criteria/approach  to  trades 

>  Unclear,  incomplete  and  inconsistent  organizational  roles 
/responsibilities/authorities  of  program  functionals  and  IPTs;  charters,  chairs, 
members,  products  -  link  to  WBS,  EVMS,  TPMs. 

>  Lack  of  clarity  on  approach,  products,  responsibilities  for  CM  [i.e.,  Technical 
Baseline  Management  -  when  does  Government  take  control?  CCB  structure? 

>  Lack  of  complete  and  specific  information  on  Technical  Reviews  -  approach,  chair, 
tailored  entry/exit  criteria,  stakeholders/independent  SMEs. 

>  Inadequate  Integrating  SE  with  other  program  plans/processes  (e.g.,  Acquisition 
Strategy,  IMS,  EVM,  Risk  Management,  production,  sustainment/logistics) 

>  Lack  of  specifics  as  to  incentives/award  fees  for  good  SE. 

>  Generic,  not  tailored,  and  vague  SE  process  descriptions. 
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Systems  and  Software  Engineering 


SEP  Bloopers  © 


>  “Task  analyses  conducted  by  human  and  engineers  provide 
qualitative  data  to  support 

>  “Fifteen  (15)  trade  studies  are  planned  during  the  SDD  phase.  These 
trade  studies  are  undefined  at  this  time.” 

>  “Integrity  is  not  an  issue  on  the  {Program},  because  the  program  was 
put  on  contract  during  acquisition  reform.” 

>  “The  ...  Program  Manager  and  Systems  Engineer  monitor  integration 
activities  to  ensure  that  the  KPPs  and  the  KSAs  are  not  achieved.” 

>  “The  ...communications  are  intended  to  support  both  the  internal 
communications  capabilities  and  external  interfaces  between  the 
{Program}  and  the  rest  of  the  world” 

>  “The  {Program}  technical  reviews  conducted  during  the  PD  and  O&S 
phases  are  chaired  by  a  competent  person.” 


Sources:  extracts  from  various  program  SEPs 
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System  Acquisition  Issues  Identified 
and  Captured 


Systems  and  Software  Engineering 


Next  a  summary  of  recent  issues  identified  as  they  relate 
to  SE  activities: 


•  SE  and  SWE  issues  from  NDIA-SE  Workshops 

•  Program  Support  Reviews  systemic  analysis  findings 
from  ODUSD(A&T)SSE/Assessments  and  Support 


We  will  list  and  compare  similarities  across  these  findings  and  SEP 
observations  as  they  relate  to  the  SE  processes 
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NDIA-SE  Top  5  Systems  Engineering 
Issues 


Systems  and  Software  Engineering 


•  Key  SE  effective  practices  inconsistently  applied  across  all  phases  of  the  LC 

•  Insufficient  SE  applied  early  in  program  life  cycle,  compromising  foundation  for 
initial  requirements  and  architecture  development 

•  Requirements  not  always  well-managed,  e.g.,  ineffective  translation  of  needed 
capabilities  into  executable  requirements  to  achieve  program  success 

•  Quantity  and  quality  of  SE  expertise  insufficient  to  meet  demands  of 
government  and  defense  industry 

•  Collaborative  environments,  e.g.,  SE  tools,  inadequate  to  effectively  execute 
SE  at  joint  capability,  system  of  systems  (SoS)*,  and  system  levels. 

*  Significant  note:  issues  relative  to  evolving  acquisition  strategies  and  environments 
were  also  a  common  theme.  Although  task  group  ultimately  decided  to  capture  these 
aspects  as  comments  distributed  across  above  5  major  issues,  SoS  issues  are 
significant  and  in  aggregate  could  be  considered  a  “6th  issue”  added  to  this  list. 

*  From  NDIA-SE  task  group;  2006  Full  report  can  be  found  at  http://www.ndia.org 
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NDIA-SE  Top 


7  Software  Issues 


Systems  and  Software  Engineering 


1 .  The  impact  of  requirements  upon  software  is  not  consistently  quantified 
and  managed  in  development  or  sustainment. 

2.  Fundamental  system  engineering  decisions  are  made  without  full 
participation  of  software  engineering. 

3.  Software  life-cycle  planning  and  management  by  acquirers  and  suppliers  is 

ineffective. 

4.  The  quantity  and  quality  of  software  engineering  expertise  is  insufficient 
to  meet  the  demands  of  government  and  the  defense  industry. 

5.  Traditional  software  verification  techniques  are  costly  and  ineffective  for 
dealing  with  the  scale  and  complexity  of  modern  systems. 

6.  There  is  a  failure  to  assure  correct,  predictable,  safe,  secure  execution 
of  complex  software  in  distributed  environments. 

7.  Inadequate  attention  is  given  to  total  life  cycle  issues  for  COTS/NDI  impacts 
on  lifecycle  cost  and  risk. 


Source:  NDIA  Top  Software  Issues  Workshop;  August  2006;  DOUSD(A&T)SSE/SSA 
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Top  10  Emerging  Systemic  Issues 

[from  ODUSD(A&T)  Program  Support  Reviews;  SSE 
Directorate, 


Systems  and  Software  Engineering 


1.  Management 

•  IPT  roles,  responsibilities,  authority,  poor  communication 

•  Inexperienced  staff,  lack  of  technical  expertise 

2.  Requirements 

•  Creep/stability 

•  Tangible,  measurable,  testable 

3.  Systems  Engineering 

•  Lack  of  a  rigorous  approach,  technical  expertise 

•  Process  compliance 

4.  Staffing 

•  Inadequate  Government  program  office  staff 

5.  Reliability 

•  Ambitious  growth  curves,  unrealistic  requirements 

•  Inadequate  “test  time”  for  statistical  calculations 

6.  Acquisition  Strategy 

•  Competing  budget  priorities,  schedule-driven 

•  Contracting  issues,  poor  technical  assumptions 

7.  Schedule 

•  Realism,  compression 

8.  Test  Planning 

•  Breadth,  depth,  resources 

9.  Software 

•  Architecture,  design/development  discipline 

•  Staffing/skill  levels,  organizational  competency  (process) 

10.  Maintainability/ 
Logistics 

•  Sustainment  costs  not  fully  considered  (short-sighted) 

•  Supportability  considerations  traded 

Major  contributors  to  poor  program  performance 


SE  Issues  Example  Mapping  -  lanagemen 

(*  SE  processes  with  top  issues  -  authors  own) 
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ISO/IEC  15288 

SE  issues 

SW  issues 

PSR  findings 

SEP  observations 

Project 

Planning* 

Inconsistent 

SE 

practices; 
insufficient 
early  SE 

Ineffective  life 
cycle 
planning, 
estimation 

IPT  roles/  responsibilities; 
non  rigorous  SE 
approach;  compressed 
schedule  driven;  coupling 
IMP/IMS/WBS 

Incomplete;  inconsistent;  unclear 
responsibilities 

Project  Asses  - 
ment  &  Control* 

Inadequate 

tools 

Ineffective 

management 

Poor  communication 

see  others 

Measurement 

Requirements 

Usually  little  specifics  (e.g.,  TPM 
allocations  to  IPTs) 

Decision 

Management 

Key  decisions 
made  w/o  SW 
participation 

Little  details  on  who  &  how  (other  than 
IPTs  communicate) 

Risk 

Management* 

Inadequate  re 
COTS/ND  1 

SE  -  SW  integration 

Lack  of  details,  responsibility,  risk 
mitigation 

Configuration 

Management 

All  key  baselines  not  clearly  defined; 
nor  when  transition  to  Government 

Info.  Mgt. 

Acquisition  & 
Supply 

Ineffective 

management 

Lack  of  SE  specific  incentives/award 
fees  (sometimes  too  much 
responsibility  deferred  to  Prime) 

Project- 

Enabling 

processes* 

Insufficient 

SE  skills; 
inadequate 
collaborative 
environment 

Insufficient 

SW  engr. 
expertise; 
process 
compliance 

Inexperienced,  inadequate 
staff 

SE  Issues  Example  Mapping  -  Technical 

(*  SE  processes  with  top  issues  -author’s  own) 


Systems  and  Software  Engineering 


ISO  15288 

SE  issues 

SW  issues 

PSR  findings 

SEP  observations 

Stakeholder 

Requirements 

Definition 

Not  well  managed 
or  translated 

Poor  definitions 

Requirements 

Analysis* 

Not  well  managed 
or  translated 

Not  consistently 
quantified  & 
managed 

Unrealistic  reliability  goals; 
test  time 

Architectural 

Design 

Poor  technical 
assumptions;  architecture 
design  &  development 

Implementation 

Integration 

Lacks  specifics  on  who  & 
what  re  integrating 
elements,  interfaces 

Verification 

Costly  &  ineffective 
techniques  for 
scale/  complexity 

Test  planning  breadth  / 
depth  /  resources 
inadequate 

(details  would  be  from  in 
TEMP  reviews) 

Validation 

failure  to  assure 
proper  execution 

Technical  Reviews  lack 
criteria,  clear  roles, 
participants 

Transition 

Lacks  details  for 

Production  &  Deployment 

Operations; 

Maintenance; 

Disposal 

Sustainment  / 
supportability  lightly 
considered 

Lacks  details  on  O&S 

Total  SE  Capability  vs.  Project 
Performance 
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Lower 
Capability 
(x  *2.5) 
N  =  13 


Moderate 
Capability 
(2.5<x<3.0) 
N  =  17 


Higher 
Capability 
(x  *3.0) 
N  =  16 


Best 

Performance 
(x  >  3.0) 

Moderate 
Performance 
(2. 5  <  x*3.0) 

Lower 

Performance 
(x  <  2.5) 


Gamma  =  0.32 
p  =  0.04 


Source:  “A  Survey  of 
Systems  Engineering 
Effectiveness” 
by:  NDIA  SEE  Committee 
@INCOSE  -  Orlando 
Chap,  .Geoff  Draper, 
February  28,  2008 


Projects  with  better  Systems  Engineering  Capabilities  deliver  better 
Project  Performance  (cost,  schedule,  functionality) 
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Wrap  Up 


DoD  and  NDIA  are  already  addressing  some  key  issues, 
e.g., 

>  SE  technical  planning  guidance  to  program  SE  WIPTs 

>  Defense  Acquisition  Program  Support  (DAPS) 
Methodology  update  (for  PSRs) 

>  SoS  guide 

>  Engineering  for  System  Assurance  guide 

>  DT&E  guide 

>  Updated  DAG  based  on  new  DOD  Acquisition 
Management  Policy  (DoDI  5000.2)  and  ISO  15288, 

>  Some  SW  Engineering  focus  areas  (WBS,  estimation,...) 

>  University  affiliated  SE  research  program 

>  DAU  SE  courses  and  Certification 
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Backups 


Systems  and  Software  Engineering 


>  SE  “V”  for  MS  B:  SD&D  phase 

>  ISO/IEC  -12207  Software  life  cycle  process 

>  IEEE  -  1220:  SE  Process 

>  EIA  -  632:  Processes  for  Engineering  a  System 

>  INCOSE  SE  Handbook  -  Planning  Process  example 

>  DoD’s  Acquisition  Life  Cycle:  Old  vs  New 

>  Early  SE  Initiation 

>  Acronyms 

>  References 

>  Links 
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ISO/IEC  12207:2008  :  Software  life  cycle 

processes  (*changes  in  red) 


Technical 

Processes 

Stakeholder 

Requirements  Definition 
Process* 

System  Requirements 
Analysis  Process* 

System  Architectural 
Design  Process* 

Implementation  / 

Process* 

System  Integration 
Process* 

System  Qualification 
Testing  Process 

SW  Installation 
Process 

SW  Acceptance  Support 
Process 

SW  Operation 
Process* 

SW  Maintenance 
Process* 

SW  Disposal 
Process* 

SW  Implementation 
Processes 


SW  Implementation 
Process* 


SW  Requirements  Analysis 
Process 


SW  Architectural  Design 
Process 


SW  Detailed  Design 
Process 


SW  Construction 
Process 


SW  Integration 
Process* 


SW  Qualification 
Process 


Source:  IEEE  Std  12207-2008 


SW  Support 
Processes 


SW  Documentation  Mgt.. 
Process 


SW  CM 
Process* 


SW  QA 
Process 


SW  Verification 
Process* 


SW  Validation 
Process* 


SW  Review 
Process 


SW  Audit 
Process 


SW  Problem  Resolution 
Process 


Systems  and  Software  Engineering 


SW  Reuse  Processes: 

*  Domain  Engineering 

*  Reuse  Asset  Mgt.. 

*  Reuse  Program  Mgt.. 


Note:  Agreement , 
Organizational 
Enabling,  and  Project 
processes  are 
essentially  the  same 
as  ISO/IEC  15288 

*  Implies  SI/I/  similar  to  SE 
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IEEE  1220:  SE  Process  -  2005 


Systems  and  Software  Engineering 


Clause  4  -  General  Requirements 

1.  SE  process 

2.  Polices  &  procedures  for  SE 

3.  Planning  the  technical  effort: 

4.  Development  strategies 

5.  Modeling  &  prototyping 

6.  Integrated  repository:  data,  tools. 

7.  Integrated  data  package:  HW,  SW 

8.  Specification  tree 

9.  Drawing  tree 

10.  System  breakdown  structure 

11.  Integration  of  the  SE  effort: 
concurrent  engr.,  Int.  teams 

12.  Technical  reviews 

13.  Quality  management 

14.  Product  and  process 
improvement:  re-engineering, 
self-assessment,  Lessons 


Note:  Standard  includes  detailed  flows 
for  each  activity;  and  an  example 
I —  SEMP  table  of  contents 


Clause  6  -  The  SE  Process 


*  Require¬ 
ments/ 
Functional 
/Design 
trade 
studies  & 
assess¬ 
ments 
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EIA  -  632:  Processes  for  Engineering  a 
System  (1999;  reaffirmed  2003) 
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(Source:  IN  COSE  SE  Handbook  v2) 


Technical  Management 

\ 

Planning 

Assessment 

Control 

v  Process 

Process 

Process 

) 

Plans , 
Directives, 
&  Status 


Acquisition 

Request 


Note:  provides 
detailed  activities 
and  outcomes  for 
each  process 


Systems 
Analysis 
V  Process 


c 

Acquisition  and  Supply 

A 

Supply  Process 

V 

Acquisition  Process 

J 

l  CONOPS  &  Requirements 


C 

System  Design 

\ 

\ 

Requirements  Definition  Process 

Solution  Definition  Process 

_ J 

Architectures/DesiQ 

y 

/ 

Product  Realization 

V 

Implementation  Process 

Transition  to  Use  Process 

J 

1  Products 

Outcomes 

& 

Feedback 


System 

Products 


Technical  Surmort 


Requirements 

Validation 

Process 


Product 

Verification 

Process 


Product 

Validation 

Process 


INCOSE  SE  Handbook  -  Planning  Process  Example 


Systems  and  Software  Engineering 


Figure  5-2  Context  Diagram  for  the  Project  Planning  Process 


Note:  Handbook  is  based  generally  on  ISO  15288: 
2002;  will  be  updated  to  be  in  sync  with  2008 
version 


Source:  INCOSE  SE  Handbook  v3. 1 
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New  Life  Cycle  Old  Life  Cycle 


DOD’s  Acquisition  Life  Cycle:  Old  vs  New 
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User  Needs  & 
Technology  Opportunities 


Process  entry  at  Milestones  A ,  B,  or  C 
Entrance  criteria  met  before  entering  phase 

Evolutionary  Acquisition  or  Single  Step  to  Full 
Capability 


/a\ /b\' 


(Program 

Initiation) 
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Refinement 

Technology 

Development 

System  Development 
&  Demonstration 

Production  & 
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Support 
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<  >  Readiness 
Review 

LRIP/IOT&E  /S  Decision 

>r  Review 

Pre-Systems  Acquisition 


Systems  Acquisition 


Sustainment 


User  Needs 


Technology  Opportunities  &  Resources 


The  Materiel  Development  Decision  precedes 
entry  into  any  phase  of  the  acquisition  framework 

Entrance  criteria  met  before  entering  phase 

Evolutionary  Acquisition  or  Single  Step  to 
Full  Capability 


/a\ /b\' 


(Program 

Initiation) 
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IOC 


FOC 


Materiel 

Solution 

Analysis 

Technology 

Development 

Engineering  and 
Manufacturing 
Development  & 
Demonstration 

Production  & 
Deployment 

Operations  & 
Support 

^  Materiel 
>  Development 
X  Decision 

>\Post-CDR 

Assessment 

LRIP/IOT&E  A  Decision 

V  Review 

Pre-Systems  Acquisition^^^y 


Systems  Acquisition 


x\  Sustainment" 


Decision  Point  A=  Milestone  Review 


Source:  DUSD(A&T)  SSE 


SE  Provides  a  Technical  Foundation 

for  Acquisition  (based  on  new  DoD  Acquisition  Policy) 


MDD 

Business  i 
Decisions  Agreement 
to  pursue 
a  material 
solution 


Engineering 

Support 
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Preferred 

System 

Analysis 


MS 
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Selection 
of  a 

preferred 

solution 


Preferred 

System 

Concept 


MS 
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Technology 

Maturation 

And 

Prototyping 


Formal 

Program 

Start 


System  i 
Level 
Specs 


Preliminary 

Design 


Completed 
Design 


Material  Solution  Analysis 


Technology  Development 


Systems  Engineering  is  most  effective  when  it 
is  initiated  early  to  start  a  program  right! 
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Systems  and  Software  Engineering 


Source: 

National 

Research 

Council 

“Pre-Milestone 
A  and  Early- 
Phase 
Systems 
Engineering” 
Jan  2008 


Acronyms/Definitions 


Systems  and  Software  Engineering 


>  A&T  Acquisition  and  Technology  [@ODUSD] 

>  ANSI  -  American  National  Standards  Institute 

>  CMP  -  Configuration  Management  Plan 

>  DAG  -  Defense  Acquisition  Guidebook 

>  DAU  -  Defense  Acquisition  University 

>  DoD  -  U.S.  Department  of  Defense 

>  DoDI  -  DoD  Instruction 

>  EIA  -  Electronic  Industries  Alliance 

>  FRP  -  Full  Rate  Production 

>  GEIA  -  Government  Electronics  and  Information  Technology  Association 

>  IEC  -  International  Electrotechnical  Commission 

>  IEEE  -  Institute  for  Electrical  and  Electronics  Engineers 

>  INCOSE  -  International  Council  on  Systems  Engineering 

>  IOT&E  -  Integrated  Operational  Test  &  Evaluation 

>  IMP/IMS  -  Integrated  Master  Plan/Integrated  Master  Schedule 

>  ISO  -  International  Standards  Organization 

>  IOC  -  Initial  Operating  Capability 

>  IT  -  Information  Technology 

>  LRIP  -  Low  Rate  Initial  Production 

>  NDIA  -  National  Defense  Industries  Association  [SE  division] 

>  PM  I  -  Project  Management  Institute 

>  PSR  -  Program  Support  Review 

>  QA- Quality  Assurance 

>  QMP  -  Quality  Management  Plan 

>  RMP  -  Risk  Management  Plan 

>  SE  -  Systems  Engineering 

>  SEE  -  SE  Effectiveness 

>  SEI  -  Software  Engineering  Institute  [@Carnegie  Mellon  U.] 

>  SEMP  -  SE  Management  Plan 

>  SEP  -  Systems  Engineering  Plan 

>  SoS  -  System  of  Systems 

>  SSCI  -  Systems  and  Software  Consortium 

>  SSA  -  Software  Engineering  and  Systems  Assurance 

>  SSE  -  Systems  &  Software  Engineering  Directorate  [@ODUSD  (A&T] 

>  SW  -  Software 

>  SWE  -  Software  [SW]  Engineering 
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Some  SE  Related  Process  References 


Systems  and  Software  Engineering 


>  CMMI®-ACQ:  SEI/CMU,  11/07 

>  Defense  Acquisition  Guidebook,  Chapter  4  -  Systems  Engineering; 
Defense  Acquisition  University,  2004  (soon  to  be  updated) 

>  EIA/IS  -  632:  1998  -  Processes  for  Engineering  a  System 

>  IEEE  1220:  2005  Application  and  Management  of  the  Systems 
Engineering  Process 

>  IEEE/EIA  12207:  2007  (adopted  ISO/IEC  12207:2007) 

>  INCOSE  Systems  Engineering  Handbook,  v3.1;  8/2007 

>  ISO/IEC  15288:  2007  System  Engineering  -  System  Life  Cycle 
Processes 

>  NDIA  SE  Effectiveness  (SEE)  Study;  2008 

>  Understanding  and  Leveraging  a  Supplier’s  CMMI  Efforts; 

ODUSD(A&T)SSE,  2007  34 


Other  References  and  Links 


Systems  and  Software  Engineering 


Some  References: 

>  “Special  Feature:  Standards  in  Systems  Engineering”,  INCOSE  Insight ,  April  2007 
(see  particularly  K.  Crowder,  D.  Kitterman,  T.  Doran,  R.  Harwell,  and  S.  Arnold  articles) 

>  CMMI  -  Next  Steps;  Kristen  Baldwin,  ODUSD(A&T)SSE/SSA,  CMMI  technology 
Conference;  November,  2007 

>  “Harmonization  of  Systems  and  Software  Engineering  Processes”,  James  W.  Moore; 
Mitre;  June,  2007,  brief  for  ASQ-DC  meeting 

>  Issue  on  Systems  Engineering,  CROSSTALK,  STSC,  October  2007 

Links: 

>  ANSI/EIA-632:  http://www.qeia.orq/index.asp?bid=552 

>  CMMI:  http://www.sei.cmu.edu/cmmi/ 

>  DAU-DAG:  http://akss.dau.mil/daq/ 

>  IEEE  -  http://www.ieee.org/web/standards/home/find.html 

>  INCOSE  -  Standards  site  http://www.incose.org/practice/techactivities/standards.aspx 

>  INCOSE  Guide  to  SE  BoK:  http://q2sebok.incose.org/ 

>  ISO:  http://www.iso.org/iso/iso  cataloque.htm  Nook  for  ISO/IEC  15288.  122071 

>  NDIA-SE:  http://www.ndia.orq/Template.cfm?Section=Divisions  [then  select  SE] 

>  ODUSD  (A&T)  SSE:  http://www.acq.osd.mil/sse/ 
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A  CONTINUOUS  PROCESS 

VIEW  OF  SYSTEMS 
ENGINEERING  FOR  THE 
SUSTAINMENT  PHASE 

By:  Paul  Ratke 
US  Air  Force 
Oklahoma  City  Air  Logistics  Center 


HI'S  J  U  ST  M  E 


The  views  and  opinions  presented  here  are 
the  author’s.  They  are  not  necessarily 
representative  of  any  official  position  of  the  US 
Government  or  any  of  it’s  organizations. 


What’s  this  about? 

0  The  acquisition  lifecycle  framework 
shows  sustainment  as  a  small  and 
somewhat  linear  part  of  the  lifecycle. 

0  An  alternative  view  is  useful  to  aid 
understanding  of  the  sustainment 
phase. 


Where  is  this  going? 


0  Why  have  alternative  views? 
0  What  are  the  current  views? 
0  What  is  the  alternative  view? 


0  What  do  we  do  with  it? 


Why  have  alternative  views? 


0  Different  vantage  points 
0  “Pre-sustainment”  phases  emphasized 
0  There  is  much  to  be  gained 


Why  have  alternative  views? 

(Different  Vantage  Points) 

©  People  have: 

•  Different  backgrounds 

•  Different  situations 

•  Different  understanding 

©  Example: 

•  Accident  witnesses 


Why  have  alternative  views? 

(Different  Vantage  Points) 


What  Did  YOU  See? 


Why  have  alternative  views? 

(Different  Vantage  Points) 


What  Did  YOU  See? 


Why  have  alternative  views? 

(Different  Vantage  Points) 


What  Did  YOU  See? 


Why  have  alternative  views? 

(Different  Vantage  Points) 


What  Did  YOU  See? 


Why  have  alternative  views? 

(Different  Vantage  Points) 

0  Look  at  all  sides  of  the  issue 
0  Different  views  ring  with  different  people 

•  Different  backgrounds 

•  Different  situations 

•  Different  understanding 

0  Both  of  these  can  clarify  the  common  view 


Why  have  alternative  views? 

(Different  Vantage  Points) 


2x3 


®  Example. 

•  1.  Memorize  it.  —0 

•  2.  Explain  it  as  2+2+2  or 

•  3.  Explain  it  as  3+3  or 


M  H 


4.  Explain  it  as 


Why  have  alternative  views? 

(Different  Vantage  Points) 

©  Systems  Engineering  is  loaded  with 
many  view  points 

•  Role  (Government,  Contractor) 

•  Mission 

•  Etc 

•  Etc 


Position  in  Life  Cycle! 


Why  have  alternative  views? 

(“Pre-Sustainment”  Phases  emphasized) 

©  LCSE,  “Life-cycle 
©  In  vogue  to  say  it 
©  Hard  to  grasp! 

©  Harder  to  do!! 

©  There  are  reasons  to  focus  on  early  phases 
©  Where  is  SUSTAINMENT?  -  an  example 


Why  have  alternative  views? 

(“Pre-Sustainment”  Phases  emphasized) 

©  DAU  SYS302 

•  Capstone  SE  Leadership  Course 

•  80  Hrs,  Six  Team  Exercises  -  Nice  Course 
o  Entry  into  Acquisition 

o  Requirements  Development 
o  Technical  Organization 
o  Technical  Baselines  and  Earned  Value 
o  Technical  Reviews 
o  Transition  to  Production 


Why  have  alternative  views? 

(“Pre-Sustainment”  Phases  emphasized) 

0  Reasons  — 

0  Start  at  the  beginning 

•  Hard  to  start  at  end,  etc 

0  Concept,  Design,  Mfg-$1,  $10,  $100 


0  These  are  good  reasons! 


Why  have  alternative  views? 

(Much  to  be  gained) 


Q 


SYSTEM  DLYIILCPMEN  T  APD 


OEMQhGTRAli^fHASt 


C0HCEP1  REFINEMENT  AND 

re£MHOi.**Y  tevaopMeN  i 

PHASE 


DISPOSAL  PHASE 


PflODUCTBH MB  DEPLOYMENT  PHASE 


SUSTAWMFNT  PHASE 


Program  Life-Cycle  (Illustrative) 


Why  have  alternative  views? 

(Much  to  be  gained) 


CONCEPT  REFINEMENT  AND 
fiCMWOLOC^Y  DtVULOPMCNl 


PJHMXJCTKlHAftD  DEPLOYMENT  PHASE 

SUSTAWMENT  PHaE  I 


Program  Life-Cycle  (lllustr 


Why  have  alternative  views? 

(Much  to  be  gained) 


Why  have  alternative  views? 

(Much  to  be  gained) 


III 

r~ 


Why  have  alternative  views? 

(Much  to  be  gained) 

0  $1,  $10,  $100  -  Concept,  Design,  Mfg 

•  What  about  cost  in  operation?  -  1000? 

•  Fuel,  Parts,  Mission  loss 

0  Future  program  avoidance  $! 

•  Ability  to  Modify  -  ‘Modifiability’  as  an  ‘-ility 

•  B-58&  F-111  -  avoid  B-1? 

•  KC-135  -  avoid  KC-next? 


Where  is  this  going? 


0  Why  have  alternative  views? 
0  What  are  the  current  views? 
0  What  is  the  alternative  view? 


0  What  do  we  do  with  it? 


What  are  the  current  views? 


CJCS  3170.01  Series 
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Development 
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Oversight 
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DEPSECDEF 
Oversight 


DAG,  Chp  1 


ANNUAL  COST 


What  are  the  current  views? 
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Operating  & 
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Program  Life-Cycle  (Illustrative) 


DEFOSAL  PHASE 


DAG,  Chp  2 


What  are  the  current  views? 
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DAG,  Chp  5 


What  are  the  current  views? 


INPUTS 

■SeJVTCO  U5<*  D-ilEJ 
User 

Fiiihtr*  Report i 
5/j  Eifii  £  4>fiffCy  ^TMfyjej 
PESHE 

0/jerap  jncy  Aborts 


Operations  and  Support  Phase 


OUTPUTS 


■fjipirt  tofCDD 

tftwmmtot 

*  .TTOr*lNpS 

(o  flitted  &ystmm 
+£>ttttm  Sate!*1  Afraid 
SEP 


S«f vN*' . 


Monitor  a  nd  Collect 

Rrritw 

All  Service 

Implement  and 

u»  D;>i 41 

Field 

Analyie  Data  Id 
Determine 
Root  Cause 


Asse-ss  Pish  or 
hnp roved  System 


wO 


Detemtine 

Sv^Ihiiii  Rl  sW 
H-ii^irtl  Severity 


CoirnUivu  Action 


Develop 

Corrective 

Action 


■  Process  Change  - 
Hardwire  Support 
*  Materiel  Change 


What  are  the  current  views? 
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DAG,  Chp  5 


Where  is  this  going? 


0  Why  have  alternative  views? 
0  What  are  the  current  views? 
0  What  is  the  alternative  view? 


0  What  do  we  do  with  it? 


What  is  the  alternative 
view? 

®  Utility  Curve  view 
®  Utility  is  “the  state  of  being  useful” 

®  Consider  all  the 
operational 
“usefulness”  as  the 


system’s  “Utility” 

Consider  what  the  user 
needs  to  be  the  “Need” 


Utility 


What  is  the  alternative 
view? 


0  1  2  3  4  5 

Time 


What  is  the  alternative 
view? 


What  is  the  alternative 
view? 

©  DAG  -  Chp  5  indicates  iterative 
monitoring 

©  SE  not  once  and  done 

©  When  Utility  does  not  match  Need 

•  Either-  Effect  a  ‘non-material’  change 

•  Or-  Begin  the  modification  process 

©  Need  an  iterative  engine  during 
sustainment 


What  is  the  alternative 

vipw? 

v  i  ks  v  v  .  utility  Curve  Engine  for  Sustainment 


Is  it  Time  Yes  A 

Does 

Yes 

to 

Utility  = 

^  Iterate?  r  ^ 

Need? 
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No 

No 

: 
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Is  a 

Effect  No 

Config 

Yes 

correction 

Change 

Needed? 

Initiate 

Modification 


Where  is  this  going? 


©  Why  have  alternative  views? 
©  What  are  the  current  views? 
©  What  is  the  alternative  view? 


©  What  do  we  do  with  it? 


What  do  we  do  with  it? 


0  A  different  point  of  view? 

0  How  are  we  answering  these  questions? 
0  What  drives  the  cycle  to  start? 

0  Do  we  know  the  current  ‘Need?’ 

0  Do  we  know  the  current  ‘Utility?’ 

0  How  complete  are  our  answers  to  these? 


What  do  we  do  with  it? 


®  BIG  QUESTION 

®What  can  we  do  to 
improve  our  answers? 


Where  have  we  been? 


0  Why  have  alternative  views? 

0  What  are  the  current  views? 

0  What  is  the  alternative  view?  -  Utility  Curve 


0  What  do  we  do  with  it? 


QUESTIONS? 


THE  OPPORTUNITY  TO  MAKE  A  DIFFERENCE  HAS  NEVER  BEEN  GREATER 


Progress  Towar 


Development 


James  A.  Forbes,  PhD,  Deceased 
E.  Andrew  Long 
October  2008 


ACQUISITION  ■  FACILITIES  A  ASSET  MANAGEMENT  •  FINANCIAL  MANAGEMENT  ■  INFORMATION  &  TECHNOLOGY  ■  LOGISTICS  *  ORGANIZATIONS  ft  HUMAN  CAPITAL 


Overview  and  Outline 


•  Background 

•  Development  of  model 

•  Basic  model 

•  Intermediate  model 

•  Production/support  cost  model 

•  Summary  and  conclusions 

•  Next  steps  and  future  work 


LMI 


A 


GOVERNMENT  CONSULTING 


P  AG  E  2 


Work  Sponsored  by: 


•  Director  of  Operational  Test  and  Evaluation 

•  Deputy  Director,  Assessments  and  Support, 
Systems  and  Software  Engineering 

•  Deputy  Under  Secretary  of  Defense  (Logistics 
and  Materiel  Readiness) 


Although  the  support  of  the  sponsors  is  gratefully  acknowledged, 
positions  expressed  are  those  of  LMI  Government  Consulting,  and 
not  official  positions  of  the  sponsors. 


Study  Objective  and  Approach 


OBJECTIVE:  Mathematical  model  that  can  be  used  to  predict  the  investment 
in  reliability  required  to  achieve  a  given  amount  of  reliability  improvement 

APPROACH:  Four  sub-models  developed  in  three  phases 


Empirical  Research 

i 

Phase  II  ! 

•  Investigate  empirical 
relationships  between 
reliability  investment  and 

A.  Basic 

Model 

B.  Intermediate  1 
Model  i 

life-cycle  support  costs 

•  Develop  investment/ 

•  Develop  model  to 

reliability  improvement 

determine  effort  and 

CER 

cost  of  reliability 
engineering  process 

•  Develop  production 

and  support  cost 
model 


Phase 


Detailed  Model 


Develop  model  that  includes 
detail  on  cost  drivers  and 
impacts  of  engineering 
quality 


Nov  06 


Sep  07 


Aug  08 


Phase  I  (Empirical  Research) 


Phase  I 


Empirical  Research 


\ 

I 

/ 


Phase  II 


Phase  III 


Detailed  Model 


0 - ▲ — 

Nov  06  sep  07 


— A— 

Auq  08 


•  Developed  a  preliminary  relationship  between  investment 
in  reliability  (normalized  by  average  production  unit  cost) 
and  achieved  reliability  improvement 


•  Also,  found  that: 

-  Generally,  programs  significantly  improved  system  reliability 
with  investment,  though 

-  under-investment  in  reliability  may  be  large 

-  Reliability  goals,  although  established  and  articulated  in 
operational  requirements  documents,  do  not  appear  to  be 
driving  either  management  or  engineering  effort 


LMI 


Phase  I 


Phase  IIA  (Basic  Model) 


Sep  07  Aug  08 
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Reliability 


Phase  MB  (Intermediate  Model) 


Phase  I 


Sep  07 


Aug  08 


M 


f 


Validation 


Time 


LMI 


Phase  I 


Empirical  Research 


TAAF 


Equation  Development 


Pbrase  I 


Basic 

H 

Intermediate 

Model 

H 

Model 

Phase  I 


‘  ‘  + ±4-1  y(x)  =  [C0T  +  nb  ln(l  +  x)] 

(x)  Ma  MjL  1+xJ  cv 


M(x) 

Based  on  math  that  underlies  AMSAA’s  MPM 


LMI  cost  extension  to  AMPM 


LMI 
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COVEKPt  ME  N'T  CO  ft  SUITING 


1 

- 


Phase  I 


Empirical  Research 

i 

Pha^e  II  'I 

\ 

Basic  Intermediate 

Model  Model 

\ 

/ 

/  Phase  III 

Detailed  Model 

with  AMSAA  Data 


•  Using  25  data  points  from  eight  platforms, 
inferred  non-dimensional  TAAF  time  x  from 
the  AMPM  and  MF/M,  (neglect  xA)  ratio  of  each 
data  point 


•  Determined  LMI  model  cost  for  each  x 

-  Calibrated  model  by  adjusting  two  parameters 

•  Compared  costs  estimated  by  model  with 
AMSAA  costs 


LMI 


AMSAA  Cost  vs.  Model  Predicted  Cost 
to  Achieve  a  Given  Reliability 


AAMSA  Cost 
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Approach  to  Design  Phase  Model 


•  Adopt  A-mode,  B-mode  scheme  from  TAAF 
(and  AMSAA)  Model 

-  Assumes  process  for  identifying  and  removing  B- 
modes  is  similar  to  TAAF 

-  Engineering  labor  applied  to  PoF,  HALT,  durability, 
etc.  plays  role  similar  to  test  operation  in  TAAF 

•  Obtain  improvement  data  from  programs  that 
implemented  or  are  implementing  proactive 
tasks  (assumes  will  see  only  limited 
improvement  if  proactive  tasks  not  performed) 
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Phase  I 


Design  Period  Model  Equation 
Development 


LMI 


I  nvestment/APUC 


Phase  I 


Empirical  Research 

Basic 

Model 

Initial  Calibration  of  Design  Period  Model 


o - 4- 


Reliability  Improvement  (Mi  -  Mo)/Mo 

□  Data  ■  Model 


13  data  for  EFV,  1  datum  for  AIM-9X,  1  datum  for  MGS  Stryker 
Used  4  values  for  “goodness”  parameter 


Support  Cost  Model  (+) 


Simplified 
UAV  Example 


Investment  (or  lack 
of  investment)  in 
reliability 
improvement 


Platform  - 


Operational  time  +  ready  time 


Dependability  Operational  time  +  ready  time  +  downtime 


* 


•  N 


Platform  dependability 


Number  of  platforms 
required  to  achieve 
required  system 
dependability 


Realized  reliability 


Assume  20  hour  operational  +  ready  time. 

How  large  does  a  “flight”  of  n  platforms  need  to  be 
to  assure  at  least  one  platform  will  be  operational 
for  20  hours  with  a  given  confidence  level? 

Intend  to  buy  20  flights. 


Per  platform 
support  cost 


System 

production 

cost 


System 

support 

cost 
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20-Year  Cost,  $M 


LCC  vs.  Reliability  Investment 
Notional  UAV  Example 


LMI 
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Summary  and  Conclusions 


•  Reasonably  mature  basic  model,  17  data 
points,  all  of  which  were  historical  actuals 

•  Demonstrated  that  basic  A-mode,  B-mode 
premise  of  AMPM  can  be  extended  to  cost 
estimating 

-  TAAF  period  model  well  behaved,  but  limited  by 
use  of  estimates  rather  than  historical  actuals 

-  Design  period  model  feasibility  demonstrated, 
limited  by  use  of  estimates  and  number  of  data 
points 

•  Coupled  basic  model  to  LCC  model 


Next  Steps  and  Future  Work 


•  Continue  adding  additional  data  points  to 
basic  model 

•  In  intermediate  model 

-  Replace  TAAF  period  estimates  with  historical 
actuals  and  add  additional  platform  types 

-  For  design  period:  more  data  points,  more 
platform  types,  historical  actuals 

•  Begin  work  on  detailed  model 

•  For  all  models,  look  for  disconfirming 
evidence.  Where  do  the  models  not  work? 


LMI 
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Integrity  -  Service  -  Excellence 


2 


Findings  and  Recommendations 


U.S.AIR  FORCE 


■  Finding  #1 

Attention  to  a  few  critical  systems  engineering 
processes  and  functions  particularly  during 
preparation  for  Milestones  A  and  B  is  essential  to 
ensuring  that  Air  Force  acquisition  programs  deliver 
products  on  time  and  on  budget. 

■  Recommendation  #1 

Air  Force  leadership  should  require  that  Milestones 
A  and  B  be  treated  as  critical  milestones  in  every 
acquisition  program  and  that ...  the  “Pre-Milestone 
A/B  Checklist”  ...  be  used  to  judge  successful 
completion. 


Integrity  -  Service  -  Excellence 


3 


U.S.AIR  FORCE 


Findings  and  Recommendations 


■  Finding  #2 

Creating  a  robust  SE  process  requires  experienced 
SEs  with  domain  knowledge 

■  Recommendation  #2 

Assess  career  field  needs  and  develop  a  program 
to  address 


Integrity  -  Service  -  Excellence 


Implementation  Approach  -  2 


U.S.AIR  FORCE 


■  Established  Program  Systems  Engineer  (PSE)  shred 
under  SPRDE 

■  Active  engagement  with  SPRDE  FIPT  to  influence  DAU 
STM  courses 

■  Subject  matter  focus  has  been  realigned 

■  Provide  additional  emphasis  on  technology  transition  techniques 
and  tools 

■  Provided  70+  SMEs  to  support  competency  assessments 

■  “Science,  Mathematics,  &  Research  for  Transformation” 
(SMART)  -funded  by  OSD;  managed  by  NPS  and  ASEE 

■  Akin  to  an  undergraduate  co-op  program 

■  Also  used  to  provide  opportunities  for  graduate  students 

■  Trying  to  change  to  automatic  hire  after  award  of  degree  rather 
than  having  to  compete 


Integrity  -  Service  -  Excellence 
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Implementation  Approach  -  2 
Organic  S&E  Development 


U.S.AIR  FORCE 


■  Update  Apr  01  S&E  Strategic  Plan 


Current  &  Future  Requirements 

Goal  Areas 

Recruitment  and  retention 

Math 

initiatives 

S&T 

Education  and  training 

Acquisition 

Individual  growth  paths 

Test 

Awards  and  recognition 

Sustainment 

■  NRC  STEM  Study  (kicked  off  Aug  08;  15-month  duration) 

■  Determine  STEM  needs  of  26  functionals 

■  Fold  recommended  implementation  strategy  into  S&E 
Strategic  Plan  update 

■  RAND  S&E  Study  (SAF/AQXD  initiated) 

■  Estimating  changes  in  S&E  skills  for  emerging  technical  needs 

■  Two  time  horizons:  near  term  (5  years),  mid-term  (10-15+  years) 


Integrity  -  Service  -  Excellence 
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U.S.AIR  FORCE 


Findings  and  Recommendations 


■  Finding  #3 

Government,  FFRDCs,  and  industry  all  have  important 
roles  throughout  the  life  cycle 

■  Recommendation  #3 

Pre-A  decisions  should  be  supported  by  rigorous  SE 
processes  and  analyses  involving  teams  of  acquirers, 
users,  and  industry 


Integrity  -  Service  -  Excellence 
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Implementation  Approach  -  3 
Continuous  Capability  Planning 


U.S.AIR  FORCE 


■  Informed  Time-Phased  Requirements  Development  (ITPRD) 

■  Identify  sponsoring  MAJCOM  personnel  for  collaborative 
requirements  development 

■  Insert  acquisition  (AFMC/AFSPC/AFRL)  personnel  into  pre- 
MS/KDP-A/B  process  far  enough  in  advance  of  the  HPT  to  absorb 
context  of  program,  execute  SE  processes,  and  affect  content  of 
KPP/KSAs  and  requirements  that  go  into  AoA  planning  and 
ICD/CDD/etc. 

■  Life  Cycle  Risk  Management 

■  Comprehensive  definition  of  risk  and  risk  management;  should 
begin  at  the  earliest  stages  of  capability/program  planning  (pre- 
MS/KDP-A  capability  planning  effects),  and  continue  throughout 
the  total  life  cycle  of  the  program 

■  Modeling,  Simulation,  and  Analysis 


Integrity  -  Service  -  Excellence 
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Implementation  Approach  -  3 
Life  Cycle  Management 


U.S.AIR  FORCE 


■  High-Confidence  Criteria 

■  Strategy  should  document  multiple,  viable  trade  space 
options  for  cost,  schedule,  capability-based  performance 
requirements  and  technology 

■  Strategy  should  support  proper  phasing/synchronization  of 
requirements  with  on-  and  off-ramps 

■  Requirements  prioritized  and  properly  time  phased 
(cost/schedule) 

■  Pre-M/S-B  Risk  Management  plans  complete,  accurate, 
current  and  being  followed 


Integrity  -  Service  -  Excellence 
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Implementation  Approach  -  3 
Technology  Development 


U.S.AIR  FORCE 


■  Technology  Development  and  Transition  Strategy 

■  Extends  the  scope  of  quantitative  criteria  beyond  TRLs 

■  Includes  broader  processes  and  cross-command  forums  to 
improve  the  rigor  of  early  SE  and  contribute  to  “doable” 
requirements 

■  Increases  the  probability  that  highest-priority  shortfalls/gaps 
are  addressed 

■  Results  in  closer  alignment  between  technology  investments 
and  system  /  capability  needs 

■  Transition  Stage-Gating 

■  Provides  a  CONOPS  for  total  technology  insertion  into  the 
Acquisition  &  Sustainment  Plan 
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Implementation  Approach  -  3 
Technology  Transition 


U.S.AIR  FORCE 


■  AF  Tech  Transition  Office  (TTO)  continues  support  to 
JCTD,  QRF,  TTI  and  other  Tech  Transition  programs 

■  Tech  Transition  Program  Initiative  funded  in  FY10 
POM  ($10M/yr) 

■  Hardware  prototyping 

■  Bridge  funding  from  Tech  Demo  to  Program  POM 

■  Enterprise  interface  management  /  configuration  control 

■  Developing  R&D  Strategic  Framework  to  coordinate 
AF  policy,  programs  and  processes  to  transition 
technology  through  6.1 -6.8  to  new  program  of  record 
or  change  to  existing  program 


Integrity  -  Service  -  Excellence 


Findings  and  Recommendations 


U.S.AIR  FORCE 


■  Finding  #4 

The  organic  development  planning  function  that 
applied  pre-A  SE  to  a  number  of  successful  programs 
was  allowed  to  lapse 

■  Recommendation  #4 

A  development  planning  function  should  be 
established  in  the  military  departments  to  coordinate 
the  concept  development  and  refinement  phase  of  all 
acquisition  programs  to  ensure  that  the  capabilities  ... 
as  a  whole  are  considered  and  that  unifying  strategies 
such  as  ...  interoperability  are  addressed. 
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Implementation  Approach  -  4 


U.S.AIR  FORCE 


■  Secured  FY10  POM  funding  ($37M/yr)  for  new  PE  for 
Requirements  Analysis  &  Maturation  (RAM) 
(“Development  Planning”) 

■  Concept  Development 

■  Requirements  Analysis  Support 

■  Establishing  DP/RAM  governance  structure;  single 
point  of  entry  for  MAJCOM  DP  requests 

■  Early  SE  Guide  to  be  published  4Q  CY08 

■  Institutionalize  CCTD  and  ConSEP  in  policy 


Integrity  -  Service  -  Excellence 
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U.S.AIR  FORCE 


Implementation  Approach  -  4 
RD&E  Investment  Framework 


Transition  Assistance  --  filling  the  “Valley  of  Death” 


Basic 

Research 


Applied 

Research 


Tech 

Demo 


Corporate  S&T 


Transition 

Assistance 


Systems 
Engineering 


Prototyping 
for  Risk 


Programs  of  Record 


Prototyping 

for 


EMDD 


Procure  / 
Field  / 


Systems  Engineering 

/  Technology  Development 

System 

Integration 

Production 

Pre-Acquisition  Systems  Engineering 

Integrity  -  Service  -  Excellence 


Implementation  Approach  - 1 


U.S.AIR  FORCE 


■  Checklist  identifies  20  items  in  7  principal  areas 

■  Coverage  for  16  of  20  exists  in  current  policy 
and  guidance 

■  Conducted  informal  order-of-magnitude 
assessment  of  current  compliance  across 
practitioner  community 

■  In  process  of  identifying  process  owners  and 
key  linkages  for  each  item  needing  action 
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U.S.AIR  FORCE 


Checklist  -  Concept  Development 


CURRENT 

PROCESS 

SUPPORTING 

DOCUMENTATION 

PROCESS 

OWNER(S) 

OPR(S) 

KEY 

LINKAGE(S) 

1 

Have  at  least  two 
alternative  concepts 
been  evaluated? 

AoA  policy  in 
AFI  10-601 

•  PASEP  (pre-AoA) 

•  ASC  process  (post- AoA) 

•  Early  SE  Guide 

•  OAS,  A2/5 

•  AQR, 
AFMC/EN 

Center 

XRs 

•  AoA  and  DP 

•  ESE  guide 

•  SoS  stds  / 
practices 

2 

Can  an  initial 
capability  be  achieved 
within  ~5  years  from 
MS/KDP  B?  If  not, 
can  critical  subsystems 
(or  a  key  subset)  be 
demonstrated  within 
that  timeframe? 

New  MAIS 
programs  now 
require  IOC 
within  5  years  of 
MS  A,  per  FY08 
NDAA  Section 
811.  No  rqmt  for 
non-MAIS 

programs. 

•  Concept  SEP  (ConSEP) 

•  Transition  Plan 

•  5000.2  update  (PDR  ahead  of 
MS  B) 

A2/5  for 
DP/RAM  and 
attestation 
process 

Center 

XRs 

•  DT&E  initiative 

•  Risk 

Assessment 

•  Cost  estimating 

•  Other  enduring/ 
std  processes 

•  CCP  Guide 

3 

Will  high-risk  new 
technologies  have  been 
matured  prior  to 
MS/KDP  B?  If  not,  is 
the  risk  mitigation 
plan  adequate? 

10  USC  2366a 
requires  TRL 
~6  (defined  by 
AF  Policy 
Memo)  at  MS  B 

•  Transition  Plan 

•  ConSEP 

•  Competition  &  prototyping 
(Young  memo,  5000.2 
update) 

•  All  5 

•  DP  efforts 
and  process 
leading  to  acq 
strategies 

Center 

XRs 

with 

AFRL 

•  TD  initiatives 
(RI3,  TDTS) 

•  CCP  Guide 

4 

Have  external 
interface  complexities 
(incl.  dependencies  on 
other  programs)  been 
identified  and 
minimized?  Is  there  a 
plan  to  mitigate  risks? 

Part  of  JCIDS 
process;  SoS 

SE  guide 

•  Concept  Characterization  & 
Technical  Description 
(CCTD) 

•  CCP  process  for  developing 
options 

•  SoS  engr  (in  Early  SE  Guide) 

•  AQR  Guidance 
Memo  mandates 
CCTD 

•  A2/5  -  process 
for  developing 
option  sets 

•  AQR, 

AFMC/EN 

Center 

XRs 

•  Early  SE  Guide 

•  CCP  Guide 

•  AFMC/EN  SoS 
eng  practices 

•  All  enduring 
processes incl 
analysis 

•  TD  (RI3) 

U.S.AIR  FORCE 


Checklist  -  KPPs  and  CONOPS 


CURRENT 

PROCESS 

SUPPORTING 

DOCUMENTATION 

PROCESS 

OWNER(S) 

OPR(S) 

KEY 

LINKAGE(S) 

5 

At  MS/KDP  A, 
have  KPPs  been 
identified  in  clear, 
comprehensive, 
concise, 

understandable 

terms? 

AFI  10-601  (JCIDS 
implementation)  (at 
early  stages,  MOEs 
are  more  appropriate 
than  solution- 
focused  KPPs) 

•  ConSEP 

•  CCTD 

•  I-CDD  (to  support 
system  rqmts 
refinement  and  PDR 
prior  to  MS  B) 

•  AFMC/CC 
attestation 
point 

•  DP/RAM 
process 

Center  XRs 

•  ITPRD 
initiative 

•  Attestation 
process 

•  SE  activities 

•  LCM 

6 

At  MS/KDP  B,  are 
major  system-level 
requirements 
(including  all  KPPs) 
sufficiently  well 
defined  to  provide  a 
stable  basis  for  system 
development? 

AFI  10-601  (JCIDS 
implementation)  (at 
early  stages,  MOEs 
are  more  appropriate 
than  solution- 
focused  KPPs) 

•ConSEP 

•CCTD 

•CDD 

AFMC/CC 

attestation 

process 

SPM  and 
center  XRs 

•  DT&E 
initiative 

•  All  enduring 
processes 
including 
analysis 

•  LCM 

7 

Has  a  CONOPS  been 
developed  showing 
that  system  operation 
can  handle  expected 
throughput  and  meet 
response  time 
requirements? 

•  ConSEP 

•  CCTD 

•  I-CDD 

All 5  DP/RAM 
process 

SPM  and 
center  XRs 

•  Analysis 
framework 

•  SoS  practices 
and  standards 

•  Early  SE  - 
all  enduring 
processes 

Integrity  -  Service  -  Excellence 


17 


Checklist  -  Cost  &  Schedule, 
Performance  Assessment 


U.S.AIR  FORCE 


COST  &  SCHEDULE  SCOPING 


8 

Are  major  cost  and 
schedule  drivers  and 
risks  explicitly 
identified,  and  is  there 
a  plan  to  track  and 
reduce  uncertainty? 

•  Evaluated  within 
JROC  process  per 
JROCM  06-261. 

•  Part  of  Acq 
strategy 

Pre-A 

•  ConSEP 

•  Transition  Plan 

Pre-B 

•  SEP 

•  RMP 

•  A2/5  for 
DP/RAM 

•  Individual 
process  owners 
for  risk  &  cost 

assessment 

SPM  and 
center  XRs 
depending  on 
phase 

•  Early  SE 

•  Risk  and 
integrated 
assessments 

•  Other 
std/enduring 
processes 

9 

Have  principal 
stakeholders  accepted 
the  confidence  level 
(risk  assessment) 
associated  with  cost 
estimates? 

Cost  Estimating 
policy  &  guidance 
(POE,  ICE,  etc.) 

•  CCTD 

•  SEP 

•  RMP 

•  Risk  process  (ACE- 
AFMC/EN) 

•  Sufficiency  Rvw 
(best  of  breed  from 
Risk  Team) 

•  CE  methodology 

SPM  and 
center  XR 
depending  on 
effort/phase 

•  Risk  process 

•  Cost 
estimating 
methodology 

PERFORMANCE  ASSESSMENT 


10 

Are  models  and 
simulations  adequate 
and  appropriate  to 
validate  the  selected 
concept  and  CONOPS 
against  the  KPPs? 

•  Operational 

Context  rather  than 
“CONOPS”  per  se 

•  MOEs  at  earliest 
“checkpoints” 

•ConSEP 

•CCTD 

•SEP 

•  A2/5  (DP); 

M&S  owner  as 
enabler 

•  A2/5  from 
attestation 
perspective 

SPM  and/or 
center  XRs 
depending  on 
effort/phase; 
also  need 
M&S  owner 

•  DT&E 
initiative 

•  Analysis  Team 
products 
(M&S  activity) 

11 

At  MS/KDP  B,  do  the 
requirements  consider 
likely  future  mission 
growth  over  the  life 
cycle? 

SE/SEP  guidance 
(Address  in  updates) 

•SEP 

•Transition  Plan 

•  AFMC/CC 
attestation 

•  DP/RAM 

•  SE 

SPM  with 
insights  from 
earlier  XR 
efforts 

•  ICD  and 
I-CDD 
(validation) 

U.S.AIR  FORCE 


Checklist  -  Architecture,  Risk 


CURRENT 

SUPPORTING 

PROCESS 

KEY 

PROCESS 

DOCUMENTATION 

OWNER(S) 

OPR(S) 

LINKAGE(S) 

ARCHITECTURE  DEVELOPMENT 


12 

Has  the  system  been 
partitioned  to  define 
segments  that  can  be 
independently 
developed  and  tested? 

Architecture  views 
required  per  JCIDS 

•  ConSEP 

•  CCTD 

•  SEP 

SE  and 
DP/RAM 

Center  XRs 
and  XPM 
depending  on 
effort/phase 

•  DT&E  initiative 

•  SoS  SE 

•  ICD  and  I-CDD 
to  validate 
approach 

•  CCP  Guide 

13 

By  MS/KDP  A,  is 
there  a  plan  to  have 
information  exchange 
protocols  in  place  by 
MS/KDP  B? 

Architecture  views 
required  per  JCIDS 
(OV-3,  OV-5  and 
SV-6  should 
address) 

•  ConSEP 

•  CCTD 

•  SEP 

•  A2/5  for 
DP/RAM 
process 

•  SE  process 
including  SoS 

Center  XRs 
and  SPM 

•  SoS  practices 
and  standards 

•  early  SE 

•  DP/RAM 

14 

At  MS/KDP  B,  is  the 
program  plan 
structured  to  ensure 

that  the  contractor 
addresses  rqmts 
decomposition  / 
allocation  to 
hardware,  software, 
and  human  elements 
sufficiently  early  in 
development? 

•  SE  guidance  in 
MSBRFP 

•  WBS 

•  Acquisition  Strategy 

•  IMP/IMS 

•  SE 

•  AFMC/CC 
attestation 

SPM 

Attestation 
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Checklist  -  Risk  Assessment, 
Program  Implementation 


U.S.AIR  FORCE 


CURRENT 

SUPPORTING 

PROCESS 

KEY 

PROCESS 

DOCUMENTATION 

OWNER(S) 

OPR(S) 

LINKAGE(S) 

RISK  ASSESSMENT 


15 

Are  all  key  risk  drivers 

10-6  series? 

•  ConSEP 

SoS  engr 

Center  XRs 

•  TD  initiatives 

(including  but  not 

•  CCTD 

processes;  risk 

and  SPMs 

•  Linkage  betw 

limited  to  critical 

•  SEP 

process  (must 

depending  on 

risk,  SE  and 

technologies)  identified? 

•  TDTS 

begin  early) 

effort/phase 

SoS  eng,  Cost 

PROGRAM  IMPLEMENTATION 


16 

Does  the  program 
implementation  plan 
account  for  necessary 
and  sufficient  #  and  skill 
levels  of  organic 
(military  and  civilian), 
FFRDC,  and  support 
contractor  personnel  to 
manage  the  program? 

•  SEP  should  be  a 

resource- 
constrained  plan 

•  LCMP  should 
address. 

•  Acq  strategy 

•  Transition  Plan 

A1  -  should  be 
accounted  for  in 
Mission 
Assignment 
process  as  well  as 
during  transition 
to  a  SPO  -  all 
functionals 
(including  A2/5 
for  DP)  need  to 
be  included  in  the 

assessment 

process 

SPO  Cadre 
and  SPM 
(Center  XR, 
EN,  other 
functionals 
as  needed) 

In  work  (HCC 
definitions) 

17 

At  MS/KDP  A,  is  there 
a  plan  in  place  that 
identifies  all  necessary 
activities  and  resources 
to  reach  MS/KDP  B? 

LCMP 

Early  SE  Guide 

•  A2/5  for 
DP/RAM 

•  SE  and  SoS 
processes 

Center  XRs 
and  SPMs 
w/resource 
allocation 
process 

•  SoS 

•  SE 

•  DP/RAM 

resource 

allocation 

•  All  enduring 
processes 

V  * 

Checklist  -  Program  Implementation 


U.S.AIR  FORCE 


CURRENT 

PROCESS 

SUPPORTING 

DOCUMENTATION 

PROCESS 

OWNER(S) 

OPR(S) 

KEY 

LINKAGE(S) 

18 

Is  there  a  top-level 
system  integration  and 
test  plan? 

SEP  and  TEMP 

•  ConSEP 

•  CCTD 

•  Transition  Plan 

All  5  (DP  & 
attestation), 

PM,  SE,  SoS 

TE 

Contractor 

DT&E  and  TD 
initiatives,  SoS 
practices 

19 

At  MS/KDP  B,  are 
the  necessary  and 
sufficient  program 
management  and 
systems  engineering 
management 
personnel  in  place? 
Have  they  been 
empowered  to  tailor 
processes  and  enforce 
requirements  stability 
through  IOC? 

Usually  based  on 

PM  and  CE 
judgment  and  then 
articulated  in  SEP 
and  LCMP.  They 
are  empowered  to 
tailor  processes. 

EMA  instituted  to 
add/improve 
discipline  for 
requirements 
stability. 

•  ConSEP 

•  Transition  Plan 

A1  (Mission 

Assignment 

Process) 

SPO  Cadre 
and  SPM 
(Center  XR, 
EN,  other 
functionals 
as  needed) 

In  work  (HCC 
definitions) 

20 

Has  the  government 
attempted  to  align  the 
duration  of  the 
program  manager’s 
assignment  with  key 
milestones  and 
deliverables? 

New  policy  memo 
forthcoming 

Transition  Plan 

Mission 
assignment 
process  with 
senior  officer 

moves 

OSD 

In  work  (OSD) 

Integrity  -  Service  -  Excellence 
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Prototyping  and  Early  SE 


U.S.AIR  FORCE 


■  Basic  tenets  of  prototyping  can  help  a  program- 
to-be  directly  address  10  of  the  20  checklist 
items  --  at  least  one  in  each  of  the  7  areas 

■  A  well-crafted  prototyping  plan  can  impact  most 
if  not  all  other  items 


PROTOTYPING  AND  EARLY  SE  CHECKLIST  “BOX  SCORE” 

Concept  Development  2J '4 

Architecture  Development  2/3 

KPPs  and  CONOPS  1/3 

Risk  Assessment  1/1 

Cost  and  Schedule  Scoping  2/2 

Program  Implementation 
Strategy  1/5 

Performance  Assessment  1/2 

Integrity  -  Service  -  Excellence 
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Concept  SE  Process 


w 


U.S.AIR  FORCE 


Authorization 
to  Procee 


•  FSA  Results 

•  Capability 
Shortfall 


Trade  Space 
Characterization 


Capability 
Decomposition  / 
Analysis 

o 


Requirements 
Exploration  & 
Synthesis 


Candidate  Solution 
Set  Selection 


Trade  Space  & 
Exploratory 
Analysis 


•  CCTD 


Rqmts  Verification/ 
Capability 

Assessment 


Release 

Approval 


Final 
Concept 


Acquisition  Timeline  Review 
Analysis 
&  Verification 


n 


rr 


Cost  Analysis 
&  Verification 


Programmatic 

Analysis 


n 


Architecture 

^  — - ,  r  ^ 

Key  Subsystem 

Characterization 

Characterization 

Initial  Sk 
Concept 
Review 


Concept 

Characterization 

Review 


Solution  Set 
Technical  Analysis 


Integrity  -  Service  -  Excellence 
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U.S.AIR  FORCE 


AAduDHit  L:  Concept  ChaiAcie-iAtioa  And  Technical  Description  Fonna: 


Concept  Characterization  and  Technical 
Description  (CCTD) 

for 

Concept  Name 


D.iTE 


PrepAied  ay: 

Nam§  ofitiLTce  •'tfg.  Concept  DffwiopTttem 
OrgtmnxiisTi. . LFR1 ,  DvjNzroOcvt  erci 


Integrity  -  S e 


CCTD  Content 


>"OTE:  Subjects  in  boidlfsK*  type  lifted  minis  Table  of  Can.lEC.t5  ace  mandatory.  Design  ?md 
performance  parameters  (e.g.  “weight  power,  cool mg ,  throughput")  :"dt  identified  st.idie-s.  Analyses* 
and  or  experiments  should  be  sdecfed  an  the  basis  of  relevance  to  due  coacept  id  is  si  cm  lei  Lind  cm 
etc.  Approaches  aid  assumptions  should  ce£ect  the  andapa^Edp-.icpoiE  of  due  technical  plarnng 
(a.g  strategic  pimning.  An  A,  weapon  system  bedmalDgy  ik—diiiimi)  Des-criprive  letad  should 
be  c<3Diislec.t  with  die  amc-epfs  Level  of  maturity  tidslity  and  the  purpose  focwtiich  ihe  concept  is 
being  developed.  This  document  is  no:  expelled  -a  be  a:  ihe  level  of  a  formal  submit  cal  such  as  a 
miles-tcme  rev  :ew  product. 

TABLE  OF  CONTENTS 

1.  Mission  /  Capability  >' eel  Statement  i  C ON'OPS . _ . ._.._ _ _ 

2.  Concept  Onn'im . . . . . . . . . . . . . . 
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3.  ]  Top-Level  Archiiecrjre _ , _ _._ _ _ _ _ _ _ 
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5  -  CriticA]  Vjbsystem.Des.igL  ?nd  Sizing _ _ _ _ _ _.. 

5.5  SnpportatJity  Sustainment  Features . . . . _.. 
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5.7  Analysis  E_e  suits _ _ _ _ _ _ _ _ _ _ 

5.3  C  an cepc  Des  ign  Can  elusions  (Capabifitr  PerfonDr.nce  Description)  . 

fi.  Program  Characterization....- . . . . . . . . _.._ _ 

£.]  CriticA]  Technologies _ _ _ _ _ _ _ _ _ 

£.  2  Technology  Maturation  .-Approach _ _ _ _ _ _ 

£  3  Test  &  EvaluaiLsn  Verification  A  VaUdalum  Approach  _ _ 
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5.7  Schedule  Assumptions  _ _ _ _ _ _. _ _ _ _ 

£.  E  Cast  Analysis  Assumptions _ _ _ _ _ _ _ _ 

£.9  Oust  Estimates _ _ _ . . . . . . 

£  10  Fisk  Assessment _ _ _ _ _ _ _ _ _ 

j.  Conclusion:  .... _ _ _ _ _ _ _ _ _ _ _ _ _ _ _.. 
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"Everything  has  been  saia  before,  but  sincE  nobody  listens  ^ve  have  to  keep  going  back  and  beginning  al 

—  Andre  Gide,  Le  traite  du  Narcisse 


over  again. 


m  Support  Center 


RCM  Applied  to  the  CH-47  Chinook 

Heavy  Lift  Helicopter 


Tor  the  Warfighter  -  With  the  Warfighter 


aVNOq 


Presentation  Agenda 

•  Reliability  Centered  Maintenance  (RCM)  overview 

•  CH-47  Chinook  Introduction 

•  Application  of  RCM  Principles  to  the  CH-47D: 

-  Maintenance  Program 

-  Special  Tools  and  Test  Equipment  (STTE) 

-  Unique  Identification  (UID) 

-  Condition  Based  Maintenance  Plus  (CBM+) 


"Our  Army  at  War  --  Relevant  and  Ready". 
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What  is 

Reliability  Centered  Maintenance? 


A  zero-based,  structured 
process  used  to  identify  the 
failure  management  strategies 
required  to  ensure  an  asset 
meets  its  mission  requirements 
in  its  operational  environment  in 
the  most  safe  and  cost  effective 

manner. 


Real  honest  to  goodness  output  that 
meets  the  needs  of  the  organization 


* 


A  zero-based,  structured 
process  used  to  identify  the 
failure  management  strategies 
required  to  ensure  an  asset 
meets  its  mission  requirements 
in  its  operational  environment  in 
the  most  safe  and  cost  effective 

manner. 


A 


Zero-Based 


* 


A  zero-based,  structured 
process  used  to  identify  the 
failure  management  strategies 
required  to  ensure  an  asset 
meets  its  mission  requirements 
in  its  operational  environment  in 
the  most  safe  and  cost  effective 

manner. 


A 


Failure  Management  Strategies 


* 


A  zero-based,  structured 
process  used  to  identify  the 
failure  management  strategies 
required  to  ensure  an  asset 
meets  its  mission  requirements 
in  its  operational  environment  in 
the  most  safe  and  cost  effective 

manner. 


A 


Operational  Environment 


The  RCM  Process 


1.  Functions 

2.  Functional  Failures 

3.  Failure  Modes 

4.  Failure  Effects 

5.  Failure  Consequences 

6.  Proactive  Maintenance 
and  Intervals 


7.  Default  Strategies 


£*£CUT-/^ 


Application  of  ROM  to  the  CH-47 


*  To  reverse  the  trend  of  increasing  Operation  and 
Support  costs 

*  Chief  focus  of  maintenance  had  been  on  the 
prevention  of  failures 

-  Common  assumption  that,  in  most  cases,  equipment  “wears  out” 
and  inevitably  becomes  less  reliable  with  age 

*  With  RCM  analysis,  focus  began  to  shift  from 
preventing  failures  to  managing  the  consequences 
of  failures  as  they  affect  the  aircraft  as  a  whole. 


"Our  Army  at  War  --  Relevant  and  Ready". 
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RCM  Working  Group 

Systems  Engineer 


Test  Pilot 


Flight  Engineer/  Crew  Chief 

Mechanic 


O 

Facilitator 


Depot  Rep 

Equipment  Manufacturer 

Instructor  Pilot 


In  the  absence  of  specific  data  on  failure  rates  and  characteristics, 
intervals  are  largely  determined  based  on  service  experience. 

Often  the  most  truthful  source  of  data 


"Our  Army  at  War  --  Relevant  and  Ready". 
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£*£CUTyyd 


Maintenance  Transformation 


BEFORE  RCM 

AFTER  RCM 

200  Hour  Phase  maintenance 

400  Hour  Cycle  Service  Plan 

200  Hour  Servicing/Inspection 

•  Number  of  Phase  Maintenance  tasks  reduced  by  73% 

•  Phase  Maintenance  requires  50%  fewer  man  hours  to 
complete 

-  200  Phase:  ~67  days  downtime 

-  400  hour  Cycle  Service:  ~19  days  downtime 

-  200  hour  Cycle  Service:  ~10  days  downtime 

•  Produced  an  increase  in  readiness\ 


"Our  Army  at  War  --  Relevant  and  Ready". 
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Application  of  RCM  to  the  CH-47 


Maintenance 

Pre-Flight 


•  Eliminated  unnecessary  tasks 

-  Eliminated  Duplication  of  Effort 


Daily 

Corrosion  Inspection 
Special  Inspections 


•  200  Hour  Phase  Maintenance  Program:  Independent  Activities 

•  400  Hour  Cycle  Service:  Supportive  Activities 

-  Technical  Justification 

•  Pitot  Static  System  Check 

-  In  response  to  single  events 

•  Retorque  droop  stop  bolts  (due  to  bad  lot  of  hydrogen  embrittlement) 

-  Extended  intervals 

•  Wheel  bearing  repacking  (Extended  from  200  to  400  hours) 

-  Move  to  On-Condition  Maintenance 

•  Brake  pad  replacement 


"Our  Army  at  War  --  Relevant  and  Ready". 
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200  Flight  Hour  Phase  Maintenance  to 
400  Flight  Hour  Cycle  Service  Plan 


#  of  Tasks  Before  RCM 

200  Flight  Hour  Phase 

428 

#  of  Tasks  After  RCM 

200  Flight  Hour  Servicing 
and  Inspection 

68 

400  Flight  Hour  Cycle 
Service  Plan 

48 

Total 

116 

"Our  Army  at  War  --  Relevant  and  Ready". 
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Application  of  RCM  to  the  CH-47 


•  RCM  implementation  began  in  2004. 

-  In  August  2007,  the  CH-47  achieved  its  readiness  goal  of  75% 
Fully  Mission  Capable  (FMC)  for  the  first  time! 


"Our  Army  at  War  --  Relevant  and  Ready". 


16 


CHINOOK  (CH47D)  TOTAL  ARMY 


17 


STTE 


•  Analysis  initiated  to  determine  suitable  Basis  of  Issue 
(BOI)  to  support  Army  Transformation 

•  BOI  for  STTE  that  was  being  was  used  estimated  by 
Boeing  ~1960s 

-  Assumption  that  units  stayed  together 

-  1  of  every  applicable  Tool  was  allotted  per  25  Helicopters 

•  Needed  to  determine  suitable  BOI  so  the  Field  could 
operate  under  the  new  doctrine  of  Split  Based  Ops 


"Our  Army  at  War  --  Relevant  and  Ready". 
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Combat  Aviation  Brigade  (CAB) 


£ 


GSAB 


Attack 


General  Support  Aviation 
Brigade  (GSAB) 


10  UH-60s 


8  MEDEVAC 
UH-60s 


AVUM  Support 
Maintenance 
Company 


12  Aircraft  Company 


L 


Lift 

Aviation  Support  Brigade  (ASB) 
AVIM 

_ 

Aviation  Support  Battalion 

Personnel  (maintenance,  supply,  etc.) 


AVIM  I  AVIM 
Support  Location  llSupport  Location  2 


2  Simultaneous  Locations 


"Our  Army  at  War  --  Relevant  and  Ready". 
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STTE 


•  How  do  RCM  Principles  apply  to  STTE  (tools)? 

-  Allows  a  clear  understanding  of  the  Operating  Context 

-  Reviewed  all  maintenance  tasks  and  analyzed  tools 

•  What  tools  were  currently  recommended  versus  what  was  needed 

-  Functions,  Functional  Failures,  Failure  Modes  and  Failure 
Effects,  and  Failure  Consequences 

•  Determined  new  BOI  to  support  Army  Transformation 


"Our  Army  at  War  --  Relevant  and  Ready". 
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“The  Big  List”  Before 

422  STTE  line  items 


CH-47  STTE  After 

224  STTE  line  items 


•  Purged  obsolete  tools 

-  All  -712  engine  tools  purged  (-120) 

•  Many  items  that  were  identified  as  STTE  but  were  common  tools 

-  Dial  Indicator 

•  Purged  unnecessary  tools 

-  STVA  (Self  Tuning  Vibration  Absorber)  T railer  Adapter 


"Our  Army  at  War  --  Relevant  and  Ready". 
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£*£CUT-/^ 


ROM  Principles  Applied  to  STTE 


•  Increased  BOI  in  most  cases 

-  Example:  Actuator  Safety  Blocks  and  Rotor  Head  Lockout  Pins 
from  1  set  per  25  aircraft  to  1  set  per  aircraft 

-  Field  will  be  supplied  with  what  they  need 

•  Established  Accountability 

-  In  process  of  putting  all  STTE  on  the  MTOE  (Modified  Table  of 
Organization  and  Equipment) 

-  Means  it  must  be  inventoried  and  accounted  for 

-  Most  STTE  before  this  process  were  not  required  to  be 
inventoried. 


"Our  Army  at  War  --  Relevant  and  Ready". 
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ROM  Principles  Applied  to  STTE 


•  Acquisition  of  additional  STTE  began  2  years  ago 


•  First  two  units  equipped  in  May  and  June  2007 

•  Analysis  results  justified  an  increase  in  STTE  funding 

-  As  a  results,  the  PM  awarded  $6M  additional  funding  per  year 
for  the  next  1 0  years 

•  Funds  2  Combat  Aviation  Brigades 

•  The  real  success  is  that  the  guy  in  the  Field  has  the  tools 
he  needs!! 


"Our  Army  at  War  --  Relevant  and  Ready". 
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DoD  UID  Mandate:  Parts  Marking 


■  July  2003,  Office  of  the  Under  Secretary  of 
Defense  set  forth  policy  to  uniquely  identify 
all  legacy  and  new  asset  parts  with  a  2-D 
barcode  if  a  part  meets  1  or  more  of  5  criteria 


■  Raises  important  concerns:  how  to  mark,  where  to 
mark,  and  how  to  safely  mark 

■  CH-47:  Approximately  1,000  components  required  to  be 
marked 

■  Independent  study  performed  on  300  components 


"Our  Army  at  War  --  Relevant  and  Ready". 
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f&JiDoD  UID  Mandate:  Parts  Marking 

•  Realized  that  Parts  Marking  Decisions  in  such  a  critical  environment 
require  analysis 

•  Parts  marking  solutions  identified  using  RCM  Principles 

-  Systematic  review  of  all  Failure  Modes,  Failure  Effects,  and 
Consequences  of  each  marking  opportunity 

Systems  Engineer 

(So 

IQ)  Depot  Rep 

IQ)  Equipment 

W  Manufacturer 

0Q0 

Facilitator 

•  Incorporates  safety  and  operating  context  into  the  core  of  the  parts 
marking  decision  making. 


Test  Pilot 

Equipment 

Specialist 


Facilitated  Group  Approach 

-  Ensures  the  right  people  who 
are  sensitive  to  the  hazards  of 
the  equipment  in  its  operating 
environment  are  the  decision 
makers 


"Our  Army  at  War  --  Relevant  and  Ready". 


DoD  UID  Mandate:  Parts  Marking 


Results: 

•  -280  items  approved  for  label  marking 

•  100  items  under  review  for  marking  approval 

•  167  Direct  Part  Marking  Candidates 

•  Over  13,000  items  marked  in  the  DoD  UID  registry 


"Our  Army  at  War  --  Relevant  and  Ready". 


30 


CBM+ 


CBM  and  RCM 


■  CBM:  Powerful  Failure  Management  Strategy  that  allows 

►  Impending  failure  to  be  identified  before  the  failure  occurs  so  that  proactive 
action  can  be  taken  in  enough  time  to  manage  the  consequences  of  failure. 

■  Ex.  Measuring  brake  pads,  eddy  current,  continuous  monitoring,  etc. 

■  In  other  words,  failure  is  handled  on  the  equipment  custodian’s  terms  - 

not  the  equipment’s  terms 

■  CBM  and  RCM  are  often  mistaken  as  two  different  processes.  They 

are  not! 


DoDI  4151.22 


■  2  December  2007,  Mr.  John  Young,  the  Under  Secretary  of  Defense 
for  Acquisition,  Technology,  and  Logistics  signed  DoDI  4151.22, 

Condition  Baaed  Maintenance  Plus  (CBM*-)  for  Materiel  Maintenance 

►  Establishes  policy  for  the  application  of  Reliability  Centered  Maintenance 
(RCM)  and  Condition  Based  Maintenance  Plus  (CBM+) 

►  CBM+  is  intended  ...“to  expand  the  application  of  sensors  on  weapons 
systems  enhancing  maintenance  efficiency  and  effectiveness...” 

►  CBM  must  be  performed  correctly  in  order  to  achieve  the  DoD’s  goals. 


Functions 

Functional  Failures 

Failure  Modes 

Failure  Effects 

Failure  Consequences 

Proactive  Maintenance 
and  Intervals 


Default  Strategies 


1.  Functions 


2.  Functional  Failures 

3.  Failure  Modes 

4.  Failure  Effects 

5.  Failure  Consequences 

6.  Proactive  Maintenance 
and  Intervals 

7.  Default  Strategies 


Consideration  of  Condition  Based  Maintenance 


Start  by  Identifying  Failure  Modes  to  be  managed 


■  Physical  assets  are  managed  at  the  Failure  Mode  level 

►  Failure  Mode:  What  specifically  causes  a  Functional  Failure 

■  CH-47  example 

►  Failure  Mode:  Drive  shaft  hanger  bearing  wears  due  to  normal 
use 


Detect  Evidence  of  Impending  Failure 


■  Nearly  all  Functional  Failures  give  some  sort  of  evidence  that  failure 
is  imminent. 

►  Referred  to  as  a  Potential  Failure  Condition  or  “P” 

■  Failure  Mode:  Drive  shaft  hanger  bearing  wears  due  to  norrmi  use 

►  P{.  Vibration  that  is  detectable  via  a  continuous 

monitoring  device  applied  directly  to  the 
equipment. 

►  P2:  Vibration  that  is  detectable  by  the  crew  by  feeling 

the  drive  shafting  area  from  inside  the  cabin  in 
flight. 


Resistance  to  Failure 


P-F  Curve 


Resistance  to  Failure 


Failure  Mode:  Drive  Shaft  Hanger  Bearing  wears  due  to  normal  use 


Time 


P„:  Vibration  detectable  via 

sophisticated  monitoring  device 


100  Flight 
Hours 


P2:  Vibration  detectable  by 
the  crew  by  feeling  the 
drive  shafting  area  from 
inside  the  cabin  in  flight 

F:  Bearing  fails 

Adequate  time  to 
find  a  suitable 
landing  site 


Failure  Mode:  Drive  Shaft  Hanger  Bearing  wears  due  to  normal  use 


■  It  would  likely  be  practical  to  check  the  data  at  intervals  less  than  100  flight  hours 

■  It  would  be  equally  practical  to  feel  for  vibration  in  flight  every  30  minutes 
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P,,:  Vibration  detectable  via 

sophisticated  monitoring  device 


Time 


100  Flight 
Hours 
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P2:  Vibration  detectable  by 
the  crew  by  feeling  the 
drive  shafting  area  from 
inside  the  cabin  in  flight 

F:  Bearing  fails 

Adequate  time  to 
find  a  suitable 
landing  site 


CH-47  CBM+ 


•  49  specific  CH-47  components  selected  for  CBM+  analysis. 

•  Acknowledge  that  a  FMEA  is  required  to  properly  implement  CBM+  strategy 

•  Components  evaluated  to  identify  Failure  Modes  that  could  be  monitored. 

-  Forward  Transmission:  13  Failure  Modes  such  as 

•  Stationary  ring  gear  wears  due  to  normal  use. 

•  FWD  transmission  1st  stage  planetary  carrier  splines  wear  due  to  normal 
use. 

•  FWD  transmission  spiral  bevel  pinion  gear  wears  due  to  normal  use. 

•  Each  Failure  Mode  prioritized  for  CBM+  Implementation  based  upon 

-  Failure  consequences 

-  Frequency  of  failure 

-  Effort  required  for  implementation  (ex.  cost  of  equipment,  training,  etc.) 

•  161  Failure  Modes  were  identified  as  candidates  for  Condition  Based 
Maintenance 


"Our  Army  at  War  --  Relevant  and  Ready". 
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What  RCM  Achieved 

•  “RCM  makes  you  take  a  real  hard  look  at  what  you’re 
doing.  ” 

•  RCM  offers  results  to  better  support  the  Warfighter 

-  Reduced  Downtime  and  improved  Readiness 

-  Reduction  of  workload  to  the  soldier 

•  Relieves  unnecessary  burdens 

-  Improved  Health  of  Aircraft 

-  RCM  has  the  ability  to  change  the  maintenance  philosophy 


"Our  Army  at  War  --  Relevant  and  Ready". 
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Nancy  Regan 
256-428-8868 

NancyRegan@TheForcelnc.com 


"Our  Army  at  War  --  Relevant  and  Ready". 
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Enhancing  Systems  Engineering 
Planning  and  Practices 

Sue  O’Brien 

Acting  Director  -  RSESC 

Rotorcraft  Systems  Engineering  and  Simulation  Center 

256-824-6133 

obriens@uah.edu 

Dawn  Sabados,  Ph.D. 

Research  Engineer  III 


RSESC/ Army  — 

Cost  Sharing  Cooperative  Agreement  Initial  Goals 

■  In  August  2002  UAH  was  competitively  awarded  cooperative 
agreement  —  AMRDEC/PEO  Aviation  and  UAHuntsville 

■  Establish  a  technical  center  to  elevate  rotorcraft  knowledge  and  skill 
levels  in  Northern  Alabama  headquartered  at  UAHuntsville. 

■  Establish  degreed  SE  academic  programs 

■  Provide  System  Engineering  Support  to  Redstone  agencies 

■  Support  the  sustaining  engineering  needs  of  the  Army  Aviation 

■  Life  Cycle  Management 

■  Systems  Engineering 

■  Reliability  Centered  Maintenance 

■  Helicopter  Aerodynamics 


$1.1  Million  Investment  by  UAH 


UAHuntsville 

Rotorcraft  Systems  Engineering  and  Simulation  Center 
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RSESC 


Multifaceted  Organization  Focused  on  Applied  Systems 
Engineering 

Independent  Assessments 
Systems  Engineering  Support 

Hardware  Design,  Analysis,  Fabrication  and  Testing 
Non  Destructive  Testing  and  Evaluation 
Reverse  Engineering 
Health  Monitoring 
Damage  Tolerance 


Projects  funded  through  NASA,  PEO  Aviation,  PEO 
Missiles  and  Space,  OSD,  and  Industry 


UAHuntsville 

Rotorcraft  Systems  Engineering  and  Simulation  Center 
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Education  and  Training 


Developed  two  new  Master  of  Science  Programs 

■  Rotorcraft  Systems  Engineering 

■  Missile  Systems  Engineering 


56  Master  of  Science  Degrees  Conferred  —  Redstone 
Engineers 

2  Current  PhD  students 


Developing  two  new  AMRDEC  /  PEO  related 
curricula 


■  Reliability  Engineering 

■  Acquisition  Engineering 


UAHuntsville 

Rotorcraft  Systems  Engineering  and  Simulation  Center 


RSESC  Curriculum 

MSE— Rotorcraft  &  Missile  Systems  Eng. 

1st  Semester 

-  Selected  Topics  in  Mathematics 

-  Statistical  Methods  for  Engineers 
-Aircraft  Stability  and  Control 

3rd  Semester  4th  Semester 

-  Rotorcraft  Design  II  -  Engineering  Reliability 

-  Performance  Flight  Testing  -  System  Safety 

-  Modeling  and  Simulation  -Aviation  Systems  Simulation 


1st  Semester 

2nd  Semester 

-  Missile  Aerodynamics 

-  Missile  Design 

-  Rocket  Propulsion 

-  Graduate  Engineering  Analysis 

-Aero  Systems  Engineering 

-  Statistical  Methods 

3rd  Semester 

4th  Semester 

-  Stability  and  Control 

-  System  Simulation 

-  Performance  Flight  Testing 

-  System  Modeling  &  Analysis 

-  Reliability  Engineering  -  Integrated  Product  &  Process  Design 

UAHuntsville  5 

Rotorcraft  Systems  Engineering  and  Simulation  Center 

2nd  Semester 

-  Helicopter  Theory 
-Aerospace  Systems  Engineering 

-  Rotorcraft  Design  I 


RSESC  Labs 


■  Two  System  Engineering  Labs  w/  full  SE 
software  resources 

■  Aero  Simulation  Lab 

■  Electrical  and  Mechanical  Design  and 
Manufacture  Lab  with  a  Machine  Shop 

■  Modal  Testing 

■  Environmental  Testing 

■  Systems  Design  and  Testing  Lab 

■  NDE/NDT 

UAHuntsville 

Rotorcraft  Systems  Engineering  and  Simulation  Center 


i  ■■ 

Design  and  Analysis 


Prototype  Designs 

Independent 

Analysis 

Specialty  Analysis 


23/09/2007 


Systems  Engineering  Labs 


Fully  Integrated  SE  Lab 

Analysis  and  System  Engineering 

Software 

Integrated  with  CAD  Lab, 
Computer  Cluster,  Rapid 
Prototyping  Machines 


K  P  P 


KEY  PERFORMANCE  PARAMETERS 
[GO  NO  GO  CRITERIA] 

AC:  400Hz,  3  Phase,  1 15/200 V,  47kVA  Cont.,69kVA 
Peak 


COMPANY  1 


80-0095  0129 


9780-0073  /  -00 


DC:  28V,  210A  Cont,  500A  Peak 


HYD.:  12gpm  @  3350psig  (start),  15gpm  @  3000psig 
(service) 


GO 


GO 


PNEU.:  301b  111m  >W  30-50  psig 


Miilti-Fmiction  Aerospace  GvSE 
#ACT95 


KEY  SYSTEM  ATTRIBUTES 


Simultaneous  Operations 


Mobility 


Transportability 


Yes  No 


YES 


YES 


COMMENTS 

A/C,  DC,  Hydraulic  and 
Pneumatic 


Trailer 


LXWXH  =  13'  X  6.9’  X 
6.9’ 

Weight  =  7,700  lbs  (wet) 


Reliability 


UAHuntsville 

Rotorcraft  Systems  Engineering  and  Simulation  Center 
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Systems  Engineering  Toolkit 

& 

System  Engineering  Projects 


Revitalization  of  SE  in  DoD 


■  In  February  2004,  the  Department 
of  Defense  mandated  the  re¬ 
vitalization  of  systems 
engineering  throughout  all  the 
services 

■  All  acquisition  category  level 
programs  were  required  to  create 
system  engineering  plans  (SEP) 

■  From  this  mandate  the  Office  of 
the  Deputy  Under  the  Secretary  of 
Defense  (OSD)  created  a  SEP 
Preparation  Guide  for  all 
programs  to  follow. 


Systems  Engineering  Plan 
Preparation  Guide 


" Technical  Planning  for  Mission  Success" 

Version  2.01 
April  2008 

Department  of  Defense 

Office  of  the  Deputy  Under  Secretary  of  Defense  for 
Acquisition  and  Technology 

Systems  and  Software  Engineering 
Enteipnse  Development 


UAHuntsville 

Rotorcraft  Systems  Engineering  and  Simulation  Center 
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Problem  Statement 


■  Systems  Engineering  is  highly  complex  subject 

■  Data  is  required  in  many  engineering  fields 

■  Metrics  need  to  be  determined  to  ensure  systems 
engineering  is  performed  effectively  and 
efficiently 

■  One  method  to  collect  data  and  to  create 
metrics  was  through  a  web  based  SE  tool 


12 


Solution 


■  The  Rotorcraft  Center’s  initial  response  to  support 
PEO-Aviation  and  PEO-Missiles  and  Space  in 
enhancing  systems  engineering  planning  was  to  create  a 
checklist  to  ensure  the  requirements  for  systems 
planning  were  met  in  the  SEP. 


■  This  checklist  evolved  into  the  Systems  Engineering 
Toolkit  to  ease  the  burden  of  creating  a  SEP  and  to 
create  a  means  for  metrics,  sharing  of  information  and 
application  based  learning  to  enhance  systems 
engineering  planning. 
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Metric/Effectiveness 


■  Real  time  training 

■  Improved  means  to  determine  of  areas  of 
difficulty 

■  Clear  Indication  of  the  amount  of  time  to  create 
the  document 

■  Ability  to  collect  statistics  on  users  and  level  of 
experience 

■  Time  spent  planning  rather  than  formatting  and 
issues  with  writing  a  complex  document 
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Systems  Engineering  Toolkit 

(SET) 

The  Systems  Engineering  Toolkit  presently  assists  in 
creating  SEPs. 

It  is  anticipated  that  future  versions  will  be  composed  of 
several  systems  engineering  tools. 

The  tool  is 

■  Configuration  Controlled  with  Global  Access 

■  Web  based  for  generating  Plans  and  Technical  Documents 

■  Tailorable  to  the  Projects  Needs,  Phase  and  ACAT  Level 

■  Modular/ adaptable  system  to  many  different  documents, 
applications,  and  phases 

Available  to  DoD  agencies 


UAHuntsville 

Rotorcraft  Systems  Engineering  and  Simulation  Center 
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SEP  Preparation 


SEP  portion  of  the  tool 
is  created  from: 

■  OSD  Preparation  Guide 
DAG  Guide 

■  Briefings  from  OSD  on 
SEP  content 

Beta  Version  of  SET 
released  June  2007 

SET  Version  1.0  released 
March,  2008  based  on 
SEP  Guidance  V.  2.01 


Title  and  Coordination  Pages 
Table  of  Contents 

1.  Introduction 

1.1  Pi 

1.2  Pi 


1.3  A 
2.  Systems  1 
2.1  S; 


2.2  SI 


2.3 


2.4  T 


2.5  In 


1 

l.t 

1.2 

1.3 

2 

2.1 

2.2 

2.3 

2.4 

2.5 

3 

3.1 

3.2 

3.3 

3.4 

3.5 

4 

4.1 

4.2 

4.3 

4.4 

4.5 

5 

5.1 

5.2 

5.3 

5.4 

5.5 

6 

6.1 

6.2 

6.3 

6.4 

6.5 


M/SA&TD 

Introduction . 

Program  Descriptions  and  Applicable  Documents . 

Current  Program  Status . 

Approach  for  SEP  Updates . 

Program  Requirements . 

Capabilities,  Requirements  and  Concepts)  of  Operation . 

Other  Requirements  Linked  to  the  Preferred  Systems  Concept .. . 

Critical  Technologies . 

Technology  Maturation  Cost  /  Schedule  Constraints . 

Technology  Development  and  Evolving  Acquisition  Strategy  .... 

Technical  Staffing  and  Organizational  Planning . 

Lead/Chief  Systems  Engineer  and  T echnical  Authorities  (Functn 

IPT  Organization/Structure . 

IPT  Staffing/Functional  Skills . 

IPT  Coordination . 

Integration  with  Contractors  and  External  Organizations . 

Technology  Maturation  and  Technical  Planning . 

Technology  Maturation  Responsibility . 

Requirements  Traceability  and  Verification . 

Technical  Maturity  and  Risk . 

Mapping  the  Technical  Baseline  to  the  Preferred  System  Concep 

Updating  and  Documenting  the  Preferred  System  Concept . 

Technical  Review  Planning . 

Event-Driven  Technical  Reviews . 

Technical  Review  Management . 

Independent  Chairing  of  Technical  Reviews . 

Stakeholder  Participation  in  Technical  Reviews . 

Peer  Participation  at  Technical  Reviews . 

Integration  with  Overall  Management  of  the  Program . 

Linkage  with  Other  Program  Plans . 

Use  of  Critical  Paths  and  Technical  Reviews . 

Risk  Management  Integration . 

Test  and  Evaluation  and  Life -Cycle  Sustainment  Integration . 

Contracting  Considerations . 

ODUSD  (A&T)  Systems  and  Software  Engine ering/Enterpr is e  Development 
ATL-ED@osd.mil 


UAHuntsville 

Rotorcraft  Systems  Engineering  and  Simulation  Center 
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SEP  Preparation  Tool 


Welcome  to  the  System  Engineering  Toolkit  (SET) 

Please  log  in 

User  Name 

Password  _ _ ' 

|  Login  | 


New  User?  Help  FAQs 


Reciister  for  SET  Support  How  do  I  reciister  for  SET? 

Eligibility  DoD  All  Service  Users  Guide  How  do  I  reset  my  password? 

Branches.  Government  What  is  SET? 

Contractors 


Integrated  review  process 
Eight  types  of  users 

■  Currently  creates  SEP  into  PDF  documents,  unchangeable  only 
from  within  the  SET  preparation  tool 

■  Secure  and  controlled  access  to  programs 

Allows  multiple  users  working  on  the  same  document  at  any  time 


UAHuntsville 

Rotorcraft  Systems  Engineering  and  Simulation  Center 


Navigation 
tree  based 
on  SEP 
Preparation 
Guide  TOC 


Colored 

Status 

Indicators 


SEP  Planning  Tool 


SYSTEMS 

ENGINEERING^ 


Account:  Sue  O'Brien  (Logout:  Active  SEP:  Tool  Demo 

Available  Documents 


Generate  Reports 

Manage  Users 

Configure  SEP 
Title  &  Coordination 

Approval  Sheet 


TOOLKIT 


Mv  Page 


Based  on  OSD  Guidance 
-  #-1  Introduction 

11.1  Program  Description 
and  Applicable  Documents 

1 1 .2  Current  Program 
Status 

■  1 .3  Approach  for  SEP 
Updates 

fi2  Program  Requirements 
fi3  Technical  Staffing  and 
Organizational  Planning 
fi4  Technology  Maturation 
and  Planning 
ft5  Technical  Review 
Planning 

fi6  Integration  with  Overall 
Management  of  the  Program 


Attachments 

Images 
Acronym  List 


UAH 

VM*  UNIVtttSITV 


Webmaster  Contact  Us 

Disclaimer  FAQ 

©2007  All  Rights  Reserved  UAH 


Document 


Permissions 


Read.  Approve.  Write 


Write 


LUH 


Admin 


Tool  Demo 


Joint  Air  to  Ground  Missile  iJAbM: 


JAVELIN 


Multiple  SEPs 
and  Permission 
Levels  Available 
to  Users 


Admin 


Write  Admin.  Version  Control 


Write  Peer  Approve.  Admin 


Admin 


Admin 


Messages 


Message 

Area 


[Date 


iFrom 


|Subject 


Section  Change  Log 


Section 

Editor 

Date 

2  1  i  Table  of  KPPs 

Lisa  Liever 

22-APR-2008 

3.5. a  How  will  the  oroaram  facilitate  interaction  among  the  SE  Workina-level  Integrated  Product  Teams  (WIPT).  other 

government  organizations  and  contractors  (as  applicable)  on  technical  tasks,  activities  and  responsibilities  (e  g 

requirements  technical  baseline,  technical  reviews)?  How  will  the  program  s  organization  and  structure  facilitate  clear 

Dawn  Sabados 

19-MAR-2008 

communication  of  technical  guidance  among  these  organizations  engaged  in  SE  activities?  How  will  technical  review  entrance 

and  exit  criteria  be  handled  between  these  organizations?  How  will  the  SE  WIPT  contribute  to  and  document  the  technical 

and  management  approach? 

Account  Options 

User  Options 
Change  Password 

©2007  All  Rights  Reserved  UAH 
Patent  Pending 


Change 
Log 


SEP  Planning  T ool 


Document  Generation 


Configuration  controlled  with  automatic  change  logs 
Creates  two  types  of  PDF  documents 


Systems  Engineering  T oolkit 

■  Benefits 

■  Most  up-to-date  information 

■  Ability  to  leverage  strengths  of  other  projects /programs 

■  Uniformity  of  Process 

■  Decrease  Approval  Timeline 

■  Team-Based  SEP  Generation  =  Consistent  Execution 

■  Minimize  “Shelf-Ware” 

■  Means  to  collect  metrics  and  best  applied  practices 

■  Ten  Organizations  interested  in  or  using  the  tool 

■  TARDEC 

■  PEOIEW&S 

■  PEOC3T 

■  PEOCS&CSS 

ft  Marines  in  support  of 
JPEO  CBD 


PEO  Aviation 

PEO  Missiles  and  Space 

Joint  PEO  Chemical  &  Biological 
Defense 

NAVAIR  in  support  of  JPEO  CBD 
AMRDEC 


UAHuntsville 


UAHuntsville’s  Involvement  in  SE 


■  Partnership  created  with  AMRDEC  in  Huntsville  to  support  Project 
offices  in  SEP  development 

■  Training,  educating  and  mentoring  on  tools,  metrics  and  teaming  in 
relation  to  systems  engineering 

■  Active  member  of  the  Army  Systems  Engineering  Forum  since  its 
inception 


■  Reviewing  and  creating  workshops  in  Systems  Engineering  Planning  for 
PEO-Aviation,  PEO-Missiles  and  Space  and  NASA/MSFC 

■  Developing  processes  to  assist  in  SE  activities  for  NASA/MSFC 

■  Determining  the  effectiveness  of  SE 

■  Teaming 

■  Tailoring  for  the  SEMP  and  SE  Processes 

■  Modeling  and  Simulation  of  SE  Processes 


V  _  f  l/  ff  ^  JLM 

M  | 

WM  Wi  1 _ 

Center  for  Modeling,  Simulation  and 
Analysis 

|  Industrial  and  5Nj5tEms  EnflinEEring 
and  Engineering  management 


UAH 


College  of 


Business  Administration 


College  of  Engineering 


Industrial  and  Systems  Engineering  and  Engineering  Management 


cm 


SER-UARC 


January  23,  2008  OSD  sent  a  notice  regarding  creating  a 
Systems  Engineering  Research  (SER)  University 
Affiliated  Research  Center  (UARC). 

UAHuntsville  partnered  with  Stevens  Institute  of 
Technology,  Univ.  of  Southern  CA  and  14  other 
universities 

Two  initial  tasks  have  been  identified  that  RSESC  will  be 
involved  in 

■  SE  Effectiveness 

■  Evaluating  Methods,  Processes  and  Tools  (MPTs) 


U.S.  Department  of  Defense 

Office  of  the  Deputy  Under  Secretary  of  Defense  (Acquisition  and 
Technology)  Systems  and  Software  Engineering 


UAHuntsville 

Rotorcraft  Systems  Engineering  and  Simulation  Center 
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Summary 

■  Vast  experience  in  applied  systems  engineering  processes,  hardware 
and  software  development  to  add  value  to  overall  project  success 

■  Experience  in  Systems  Engineering  and  the  practices  of  OSD  and 
NASA 

Utilizing  graduate  and  undergraduates  on  research  projects  to 
combine  theory  with  practical  applications  and  to  help  mentor 
engineers  and  scientists  entering  in  the  workforce 

■  Willing  to  partner  with  other  universities  and  organizations  bringing 
together  the  best  assets  to  the  community 

■  Systems  Engineering  Toolkit  (SET)  is  available  to  the  DoD  PM 
offices  and  NASA 

UAHuntsviUe  and  the  Rotorcraft  Systems  Engineering 
and  Simulation  Center  is  committed  to  becoming  one 
of  the  top  research  centers  for  Systems  Engineering 

UAHuntsville  24 

Rotorcraft  Systems  Engineering  and  Simulation  Center 
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System  Concept  of  Operations: 
Standards,  Practices  and  Reality 


Nicole  Roberts,  L-3  Communications 
Robert  Edson,  ANSER 


This  document  contains  no  ITAR  controlled  Technical  Data  nor 
provides  any  ITAR  controlled  Defense  services. 


■  Problem  Statement 

■  Approach 

■  What  is  a  CONOPS? 

■  Standards 

■  Literature  Review 

■  Case  Studies 

■  Survey 

■  CONOPS  Development  Process 

■  CONOPS  Evaluation  Criteria 

■  Recommendations 


This  document  contains  no  ITAR  controlled  Technical  Data  nor 
provides  any  ITAR  controlled  Defense  services. 
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■  Inconsistent  and  ineffective  use  of  ConOps  in  the  Systems 
Engineering  life  cycle. 

•  Saw  through  initial  survey 

■  Objectives 

•  Explore  Industry  Use  of  ConOps 

•  Define  a  quality  ConOps 

•  Develop  Evaluation  Criteria  for  ConOps  goodness 


This  document  contains  no  ITAR  controlled  Technical  Data  nor 
provides  any  ITAR  controlled  Defense  services. 
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This  document  contains  no  ITAR  controlled  Technical  Data  nor 
provides  any  ITAR  controlled  Defense  services. 


A  Concept  of  Operations  (ConOps)  document  is  produced 
early  in  the  requirements  definition  process 
to  describe  what  the  system  will  do  (not  how  it  will  do  it)  and 
why  (rationale).  It  should  also  define  any 
critical,  top-level  performance  requirements  or  objectives 
(stated  either  qualitatively  or  quantitatively) 

and  system  rationale. 

(Systems  Engineering  Handbook  INCOSE-TP-2003-016-02,  Version  2a,  1  June  2004) 


This  document  contains  no  ITAR  controlled  Technical  Data  nor 
provides  any  ITAR  controlled  Defense  services. 
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Agency 

Title 

Year 

Highlights 

GEIA 

Processes  for  Engineering  a  System 

1999 

•  DoD  and  IEEE  approved 

•  No  details,  just  says  to  have  one  with  REP 

CMMI 

Guidelines  for  Creating  a  Product 
Line  Concept  of  Operations 

1999 

•  Specific  for  building  a  ConOps  for  a  large  run  one 
product  line 

•  Good  techniques  that  can  be  applied  to  system 
ConOps  also 

ANSI/ 

AIAA 

Guide  for  the  Preparation  of 
Operational  Concept 

Documents 

1992 

•  Names  it  as  an  Operational  Concept  Document 
(OCD) 

•  Most  complete  instruction  for  building  a  ConOps 

INCOSE 

INCOSE  Systems  Engineering 
Handbook 

2004 

•  Defines  what  a  ConOps  is  and  should  include 

•  Does  not  give  instruction  on  how  to  build  one 

•  Describes  what  other  phases  it  is  an  input  to 

IEEE 

IEEE  Guide  for  Information 
Technology  -  System 

Definition  -  Concept  of 
Operations  (ConOps) 

Document 

1998 

•  Gives  instruction  on  how  to  build  a  ConOps  and 
what  to  include 

•  Focused  on  software  but  can  be  used  for  other 

•  Only  one  that  says  to  include  proposed  systems  in 
this  document 

This  document  contains  no  ITAR  controlled  Technical  Data  nor 
provides  any  ITAR  controlled  Defense  services. 
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L> 

communication^ 

IntugraEud  Syslc-ms 


■  Definitions 


•  To  clearly  define  the  operational  boundaries  and  to  capture 
the  needs  of  the  user  community.  (Herald  and  Verma) 

•  Provide  stakeholder  consensus,  measures  of  effectiveness, 
standards  of  acceptance  and  system  design/architecting 
purposes.  (Ring) 

•  To  provide  verified  accurate  work  process  information  to 
validate  and  defend  projects  and  enable  management 
decisions.  (Nichols) 

•  A  document  that  focuses  on  the  achievement,  performance 
and  basic  technological  necessities  of  the  system.  (Cakmak 
and  Gokpinar) 


This  document  contains  no  ITAR  controlled  Technical  Data  nor 
provides  any  ITAR  controlled  Defense  services. 
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■  Reviewed  6  ConOps 

■  50%  appear  to  be  satisfactory 

■  Example:  SOFIA  Science  and  Mission  Operations  Plan 

•  Focus  on  system  use 

•  List  of  key  personnel  and  their  responsibilities 

•  Use  of  system  by  personnel 

•  Facilities  information 

•  Training,  support,  logistics  and  maintenance  information 
included 


This  document  contains  no  ITAR  controlled  Technical  Data  nor 
provides  any  ITAR  controlled  Defense  services. 


■  Conducted  to  understand  how  industry  is  using  ConOps 
and  what  is  considered  a  ConOps 

■  27  Questions 

■  3  Sections 

•  Basic  Overview  of  the  individual  and  ConOps  use 

•  Questions  for  people  who  have  worked  with  ConOps 

•  Questions  for  those  who  have  been  a  ConOps  author 

■  108  responses  from  18  companies  and  organizations 

•  DoD,  L-3  Communications,  Raytheon,  Boeing,  Lockheed 
Martin,  USAF,  Bell  Helicopter,  Texas  Instruments, 
Honeywell,  General  Dynamics,  Army,  and  more 


This  document  contains  no  ITAR  controlled  Technical  Data  nor 
provides  any  ITAR  controlled  Defense  services. 


■  48%  Systems  or  Lead  Systems  Engineers 

■  From  1  month  to  54  years  in  the  industry 

■  Worked  between  1  and  100  programs  with  19.9  as  the  average 


□  System  Engineer 

■  Lead  Systems  Engineer 

□  Project  Engineer 

□  Program  Manager 

■  Other 


This  document  contains  no  ITAR  controlled  Technical  Data  nor 
provides  any  ITAR  controlled  Defense  services. 
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■  1 00%  said  they  would  find  one  useful 

■  36%  have  never  worked  a  program  with  a  ConOps 

■  Stated  ConOps  Purpose 

•  89%  -  Define  the  system  use 

•  7 1  %  -  Define  the  system  boundaries 

•  37%  -  Define  the  system 

•  28%  -  Define  system  details 

■  Program  Phases  to  be  helped  by  ConOps 

•  88%  -  Requirements  Development 

•  83%  -  System  Design 

•  70%  -  Planning  for  Test 


This  document  contains  no  ITAR  controlled  Technical  Data  nor 
provides  any  ITAR  controlled  Defense  services. 
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■  Average  number  of  programs  with  a  fully  developed  ConOps  is  4.4 

■  36%  have  never  worked  a  program  with  a  ConOps 

■  76%  of  those  who  have  worked  with  a  ConOps  ranked  them  as  a  4  or  5 

■  85%  of  those  who  worked  with  a  ConOps  had  regular  access  to  it 


This  document  contains  no  ITAR  controlled  Technical  Data  nor 
provides  any  ITAR  controlled  Defense  services. 
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■  31%  completed  by  bid  phase,  27%  by  program  start-up 

■  50%  were  not  updated  throughout  the  lifecycle 

■  76%  of  the  ConOps  were  written  and  graphical 

■  28%  of  respondents  have  been  an  author 

■  55%  of  authors  were  a  systems  or  lead  systems  engineer 

■  Customer  involved  74%  of  the  time  and  user  70%  with  1 1 
people  involved  on  average 

■  3%  of  the  time  no  one  besides  the  author  was  involved 

■  Standards  used  50%  of  the  time 

■  Average  time  to  develop  is  78  days 

■  75%  of  the  time  the  author  personally  used  the  ConOps 


This  document  contains  no  ITAR  controlled  Technical  Data  nor 
provides  any  ITAR  controlled  Defense  services. 
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■  Everyone  wants  a  ConOps  but  only  one-third  of  all 
programs  have  one 

■  Requirements  Development  and  System  Design  would  be 
helped  most  by  a  ConOps 

■  Need  qualified,  experienced  systems  engineer  developing 
the  ConOps  with  multiple  inputs 

■  Industry  is  not  utilizing  developed  ConOps  to  their 
advantage  throughout  the  lifecycle  -  Only  4%  used 
through  to  the  end 


This  document  contains  no  ITAR  controlled  Technical  Data  nor 
provides  any  ITAR  controlled  Defense  services. 


14 


■  Do  not  list  any  specifics 

■  Do  not  describe  how  a  process  or  how  a  function  should  be  performed  only 
list  the  needs 

■  Include  all  stakeholders  or  representatives  for  each  area 

■  Limit  the  group  to  less  than  fifteen  people 

■  Representatives  need  the  authority  to  make  final  decisions 

■  Have  everyone  convene  in  one  place  at  the  same  time  at  least  twice 

■  Author/moderator  needs  the  skills  to  guide  the  group  and  keep  them  on  track 

■  Get  interviews  with  all  users  not  in  the  group  then  share 

■  Limit  the  document  size  without  limiting  the  information 

■  Make  sure  the  level  of  language  is  not  too  technical  to  understand 

■  Customize.  Include  information  and  change  the  format  so  all  understand  the 
needs 


This  document  contains  no  ITAR  controlled  Technical  Data  nor 
provides  any  ITAR  controlled  Defense  services. 
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Section 

Number 

Title 

Key  Elements 

1 

Introduction 

Brief  overview 

Stakeholders 

2 

References 

3 

Problem  Statement 

-  High  level  problem  statement 

4 

Program  or  System  History 

Current  likes  and  dislikes 

Current  needs 

5 

System  Use 

Detailed  explanation  of  the  system  use  including 
Users 

External  system  interfaces 

6 

System  Boundaries 

Graphic  representation  of  the  external  system 
interfaces 

Text  explanation  of  the  details  of  each  interface 

7 

System  Environment 

Basic  system  operating  environment 

Operator  environment 

Maintainer  environment 

This  document  contains  no  ITAR  controlled  Technical  Data  nor 
provides  any  ITAR  controlled  Defense  services. 
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Section 

Number 

Title 

Key  Elements 

8 

Constraints 

Details  that  are  truly  a  must  to  be  designed  around, 
possibly  including: 

-Cost  -Schedule 

-Technologies  -Power 

-Weight  -Life  expectancy 

-Space  to  design  in  -Environment 

-Performance 

9 

System  Models 

Models  or  simulations  that  help  to  show  how  the 
system  will  be  used 

9 

System  Peripherals 

-Training 

-Supportability 

-Maintainability 

10 

Expected  Output 

-  Summary  of  what  is  to  be  done 

-  Prioritization  of  what  is  to  be  done 

-  Measure  of  effectiveness 

11 

Acronyms  and  Definitions 

This  document  contains  no  ITAR  controlled  Technical  Data  nor 
provides  any  ITAR  controlled  Defense  services. 
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■  Does  it  include  all  required  sections 

•  If  not  is  there  sufficient  reasoning  why  not 

•  If  there  are  more,  is  it  too  much  information 

■  Were  all  stakeholder  groups  represented 

■  Does  it  define  just  the  needs  and  not  the  how 

■  Does  it  include  all  standards  the  system  will  be  required  to 
adhere  to 

■  Does  it  include  the  system  boundaries  and  inputs  and 
outputs 

■  Model  to  prove  that  the  system  is  possible  with  all  the 
information  given 


This  document  contains  no  ITAR  controlled  Technical  Data  nor 
provides  any  ITAR  controlled  Defense  services. 
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■  All  systems  should  have  a  Concept  of  Operations 

■  Use  the  ANSI/AIAA  standard  for  help 

■  ConOps  initial  development  should  be  done  before 
requirements  development  if  not  earlier 

■  ConOps  should  be  updated  throughout  the  program 
lifecycle 

■  ConOps  should  be  controlled  and  made  accessible  to  all 
stakeholders  working  on  the  program 

■  If  you  do  not  know  what  you  are  trying  to  get  you  will 
never  know  if  you  accomplished  it  or  not 

■  The  contractor  should  own  their  ConOps  but  ensure 
customer  involvement  during  updates 


This  document  contains  no  ITAR  controlled  Technical  Data  nor 
provides  any  ITAR  controlled  Defense  services. 
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I? 

communications 

IntugruEu^l  Syslc-ms 


Questions? 


This  document  contains  no  ITAR  controlled  Technical  Data  nor 
provides  any  ITAR  controlled  Defense  services. 
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Defining  the  future 

Counting  Software  Size: 
I  s  1 1  as  Easy  as  Buying  A 

Gallon  of  Gas? 

October  22,  2008 

-  11th  Annual  Systems  Engineering  Conference 

Lori  Vaughan  and  Dean  Caccavo 
Northrop  Grumman  Mission  Systems 
Office  of  Cost  Estimation  and  Risk  Analysis 


Agenda 


NORTHROP  GRUMMAN 


Defining  the  future 


•  I  introduction 

•  Standards  and  Definitions 

•  Sample 

•  I  mplications 

•  Summary 
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I  ntroduction 


NORTHROP  GRUMMAN 


Defining  the  future 


•  I  n  what  ways  is  software  like 
gasoline? 

•  I  n  what  ways  is  software  not 
like  gasoline? 


NORTHOP  GRUMMAN  CORPORTION© 


I  ndustry  Data  Suggests 


NORTHROP  GRUMMAN 


Defining  the  future 


•  A  greater  percentage  of  the 
functions  of  the  DoD  Weapon 
Systems  are  performed  by 
software 


Weapon  System 

Year 

%  of  Functions 
Performed  in 

Software 

F-4 

I960 

8 

A-7 

1964 

10 

F-lll 

1970 

20 

F-15 

1975 

35 

F-16 

1982 

45 

B-2 

1990 

65 

F-22 

2000 

80 

Source:  PM  Magazine 

System  Functionality  Requiring  Software 


•  I  ncreased  amount  of  software 
in  Space  Systems  and  DoD 
Weapon  Systems  -  Ground, 
Sea  and  Space/ Missile 

•  I  ncreased  amount  of  software 
in  our  daily  lives: 

-  Cars,  Cell  Phones,  iPod, 
Appliances,  PDAs... 


The  amount  of  software  used  in  DoD  weapon  systems  has  grown  exponentially 
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I  s  There  a  Standard  for  Counting  Software? 


NORTHROP  GRUMMAN 


Defining  the  future 


•  Since,  increasing  percent  of  our  DoD  systems  are  reliant 
on  software  we  need  to  be  able  to  quantify  the  software 
size 

-  Historical  data  collection 

-  Estimation  and  planning 

-  Tracking  and  monitoring  during  program  performance 

•  Software  effort  is  proportional  to  the  size  of  the  software 
being  developed 

SW  Engineering  Economics  1981  by  Dr.  Barry  Boehm 

•  "Counting"  infers  there  is  a  standard 

•  Experience  as  a  prime  integrator 

-  Do  not  see  a  standard  being  followed 

There  are  software  counting  standards  but  the  message  isn’t  out 

or  it  is  not  being  followed  consistently 
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Source  Line  of  Code  definition 


NORTHROP  GRUMMAN 


Defining  the  future 


From  Wikipedia,  the  free  encyclopedia 

"Source  lines  of  code  (SLOC)  is  a  software  metric  used  to 
measure  the  size  of  a  by  counting  the 

number  of  lines  in  the  text  of  the  program's  source  code. 
SLOC  is  typically  used  to  predict  the  amount  of  effort  that 
will  be  required  to  develop  a  program,  as  well  as  to 
estimate  or  effort  once  the 

software  is  produced." 

•  Variety  of  Software  Languages  in  which  source  code  is 
written 
-  a  to  z 

•  Ada,  Assembler,  C,  C++,  C#,  COBOL,  Fortran,  J  ava,  J  avaScript, 

Pascal,  Perl  and  SQL  to  name  just  a  few 
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Source  Line  of  Code  definition 
Physical  and  Logical 


NORTHROP  GRUMMAN 


Defining  the  future 


•  Software  Engineering  I  nstitute 
(SEI)  has  developed  checklist  as 
part  of  a  system  of  definition 
checklists  to  support 
measurement  definitions 
Software  Size  Measurement:  A 

Framework  for  Counting  Source 

Statements. 


Definition  Checklist  for  Source  Statement  Counts 


Definition  name:  Logics!  Source  State tneni§ _  Date:  8/7/92 

_ fjfrasir  definition) _  Originator:  SET 


Measurement  unit:  Physical  source  lines  j  1 

Logical  source  statements  |  /  | 

Statement  type  Definition  |  ■/  |  Data  array  | _ 1 

Includes 

Excludes 

When  a  Sine  or  statement  contains  more  than  one  type, 
classify  if  as  the  type  with  the  highest  precedence. 

1  Executable  Order  of  precedence -> 

2  Nonexecutable 

3  Declarations 

4  Compiler  directives 

5  Comments 

6  On  their  own  lines 

7  On  lines  with  source  code 

3  Banners  and  non  blank  spacers 

9  Blank  {empty)  comments 

10  B  ank  lines 

11 

12 

1 

/ 

2 

/ 

3 

4 

/ 

5 

✓ 

6 

/ 

7 

/ 

3 

/ 

•  Physical  SLOC:  One  physical  SLOC  is  corresponding  to  one  line 
starting  with  the  first  character  and  ending  by  carriage  return  or  an 
end  of  file  marker  of  the  same  line  and  which  excludes  the  blank 
and  comment  line. 

•  Logical  SLOC:  Lines  of  code  intended  to  measure  “statements" 
which  normally  terminated  with  a  semicolon  or  a  carriage  return. 
Logical  SLOC  are  not  sensitive  to  format  style  and  conventions,  but 
they  are  language  dependent. 
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Source  Line  of  Code  Samples 


NORTHROP  GRUMMAN 


Defining  the  future 


for(i=0;  i<100;  ++i)  printf("  hello");  /*  How  many  lines  of  code  is  this?  */ 

-  1  Physical  Line  of  Code  LOC 

-  2  Logical  Lines  of  Code  LOC  (for  statement  and  statement) 

-  1  Comment  Line 


for  (i=0;  i<100;  ++i) 

{ 

printfC'hello"); 

}  /*  Now  how  many  lines  of  code  is  this?  */ 

-  4  Physical  Lines  of  Code  LOC  (Is  placing  braces  work  to  be  estimated?) 

2  Logical  Line  of  Code  LOC  (What  about  all  the  work  writing  non-statement  lines?) 

1  Comment  Line  (Tools  must  account  for  all  code  and  comments  regardless  of  comment 
placement.) 

Note  the  logical  count  is  independent  of  the 
programming  style  and  conventions 
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I  mplications  of  SLOC  Counts 
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Defining  the  future 


Typical  Simplified  Software  Cost  Estimation  Formula 


Productivity 

(Size  /  Time  unit) 


Effort/  Cost 


•  Suppose  you  were  given  this  simplified  software  cost  formula  and  you 
received  data  from  two  separate  contractors  and  were  asked  to 
determine  relative  development  costs? 

•  What  would  that  impact? 

-  Size 

-  Productivity 

-  Hours 
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I  mplication  1 1  lustration  -  Historical 
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Defining  the  future 


Contractor  A 


Contractor  B 


Physical  Coordinate  Perspective  Logical  Coordinate  Perspective 


SLOC  Count 

500  KSLOC 

Effort 

2500  Person  Months 
(PM) 

Productivity 

500  KSLOC  -  2500 
PM=  200  ESLOC/PM 

SLOC  Count 

312.5  KSLOC 

Effort 

2500  (PM) 

Productivity 

312.5  KSLOC  -  2500 
PM  =  125  ESLOC/PM 

Without  understanding  the  basis  of  the  Software  SLOC 
count,  it  looks  like  Contractor  A  is  more  productive. 

Is  this  correct? 
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I  mplication  1 1  lustration  - 
Estimate  Comparison 
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Defining  the  future 


Contractor  A 


Contractor  B 


Estimated  Size 

600  KSLOC 

Historical 

Productivity 

200  ESLOC/PM 

Estimated  Effort 

3,000  PM 

Estimated  Cost 

3,000  PM  X 
$20K  =  $  60  M 

Estimated  Size 

600  KSLOC 

Historical 

Productivity 

125  ESLOC/PM 

Estimated  Effort 

4,800  PM 

Estimated  Cost 

4,800  PM  X 
$20K  =  $  96  M 
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USC  Center  for  Systems  and  Software 
Engineering 


NORTHROP  GRUMMAN 


Defining  the  future 


•  Attributes  of  a  good  code  counter 

-  Non  Proprietary 

-  Available  to  the  public 

-  Platform  independent 

-  Support  multiple  programming  languages 

-  Count  both  physical  and  logical  SLOC 

-  Limited  Public  License  or  “Copyleft"  type  agr 


Sample  1.0:: SLOC  Counting 

The  Totals 

Total  Blank  \  Comments 

1  Compiler  Data 

Exec. 

|  Number  File  SLOC 

Lines  Lines  \  Whole  Embedded  \  Direct  Dec!.  Instr.  \  of  Files  \  SLOC  Type  Definition 

33991  3855  \  8465 

19 1  250  6815 

14606 | 

336  | 

21671  CODE  Physical 

33991  3855  \  8465 

19 |  250  2775 

10667 | 

336  | 

13692  CODE  Logical 

1135  42 |  0 

0 |  0  1093 

o  1 

47  | 

1093  DATA  Physical 

Number  of  files  successfully  accessed. . . 

.  383  out  of  383 

Ratio  of  Physical  to  Logical  SLOC.... 

...  1.58 
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use  CSSE  CodeCount™ 
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Defining  the  future 


•  What  programming  languages  are  covered  today 

-  Ada  ,  Assembled s).  Jovial,  Pascal,  COBOL,  Fortran,  MUL  -  Markup 
Language,  J  ava,  C/C++,  C#,  J  avaScript,  Visual  Basic  and  Visual 
Basic  Script 

•  What  is  included  for  each  language 

-  Read  me  file 

-  Logical  Standard  (word  table) 

-  C  source  code  of  language  specific  counter 

-  Sample  input,  source  files  and  output  file 


USC  Center  for  Systems  and  Software  Engineering  (CSSE)  CodeCount™ 

suite  supports  many  languages 
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I  magine  Software  Code  Counting... 


NORTHROP  GRUMMAN 


Defining  the  future 


•  As  an  integral  part  of  your  program's  change  management 
system 

•  I  mproving  your  ability  to  perform  Root  cause  Analysis 

•  Normalized  code  counts  of  existing  software  that  are 
automatically  uploaded  to  your  historical  database 

•  A  historical  repository  of  software  size  that  could  be  used 
for  estimation  purposes  and  parametric  model  calibration 

•  I  mproving  the  representative  nature  of  Parametric  and 
Predictive  Modeling 

•  Being  consistent.... 
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Summary 


NORTHROP  GRUMMAN 


•  Recognize  underlying 
implications  of 
Physical  and  Logical 
software  sizing 

•  Assess 

appropriateness  and 
magnitude  of  code 
count  measurement 

•  Consider  widespread 
standardization  and 
integration  into 
acquisition  process 
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Defining  the  future 


Reuse  Readiness  Levels: 

A  Framework  for 
Decision  Making 

NDIA  Systems  Engineering  Conference 

October  20-23,  2008 

Steven  Wong 

Northrop  Grumman  Mission  Systems 


Topics 


NORTHROR  GRUMMAN 


•  Reuse  and  Maturity 

•  Measures  of  Maturity  -  Technology  Readiness  Levels 

-  Background 

-  Applicability  to  Software 

-  Limitations 

•  Reuse  Readiness  Levels 

-  Motivation 

-  Background 

•  SEI 

•  NASA 

-  Northrop  Grumman  Approach 

•  Reuse  Attributes 

•  Decision  Analysis  Resolution  Process 

•  Outcomes 
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To  Reuse  or  Not  to  Reuse  Software? 


NORTHROR  GRUMMAN 


•  “Good  reuse  "  economizes  time 
and  money;  ensures  quality 

-  Increased  dependability 

-  Compliance  to  standards 

-  Accelerated  development 

-  Economies  of  Scale 

-  Reduce  product  and  process  risk 


“Bad  reuse  "  introduces  risk  resulting 
in  cost  and  schedule  growth 

-  Incompatibility 

-  Obsolescence 

-  Breakage 

-  Requirements  differences 

-  Unfamiliarity 


How  can  one  make  an  a  priori  distinction 
between  good  and  bad  reuse? 
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DoD  5000. 2- R,  J  an  4,  2001 


NORTHROR  GRUMMAN 


7.5.  --  Technology  Maturity 

Technology  maturity  shall  measure  the  degree  to  which  proposed  critical 
technologies  meet  program  objectives.  Technology  maturity  is  a  principal 
element  of  program  risk.  A  technology  readiness  assessment  shall  examine 
program  concepts,  technology  requirements,  and  demonstrated  technology 
capabilities  to  determine  technological  maturity. 

The  PM  shall  identify  critical  technologies  via  the  work  breakdown  structure  (WBS)  (see 
5.3.1).  Technology  readiness  assessments  for  critical  technologies  shall 
occur  sufficiently  prior  to  milestone  decision  points  B  and  C  to  provide 
useful  technology  maturity  information  to  the  acquisition  review  process. 

The  Component  Science  and  Technology  (S&T)  Executive  shall  direct  the  technology 

readiness  assessment  and,  for  ACAT  I D  and  ACAT  I A  programs,  submit  the  findings 
to  the  Deputy  Under  Secretary  of  Defense  (S&T)  (DUSD(S&T))  with  a 
recommended  technology  readiness  level  (TRL)  for  each  critical  technology. 

I  n  cooperation  with  the  Component  S&T  Executive  and  the  program  office,  the 
DUSD(S&T)  shall  evaluate  the  technology  readiness  assessment  and,  if  he/she 
concurs,  forward  findings  to  the  Ol  PT  leader  and  DAB.  If  the  DUSD(S&T)  does  not 
concur  with  the  technology  readiness  assessment  findings,  an  independent  technology 
readiness  assessment,  under  the  direction  of  the  DUSD(S&T),  shall  be  required. 
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A  Definition 
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•  Technology  Readiness  Levels  (TRL)  are  used  to  assess  the  maturity 
of  a  practically-applied  scientific/engineering  invention  (materials, 
components,  methods,  devices,  etc.)  prior  to  its  incorporation  into  a 
system 

•  A  method  for  assessing  how  much  risk  is  potentially  involved  with 
adopting  a  technology 

•  TRLs  assume  that  a  technology  is  less  suitable  for  immediate  usage 
when  it  is  newly  invented  or  conceptualized 

•  A  technology  becomes  sufficiently  proven  (i.e.,  mature)  after  being 

subjected  to  experimentation,  refinement,  and  increasingly  i  w 

demonstrated  and  tested  in  a  realistic  environment 


•  Examples:  Hardware  TRL,  Software  TRL, 
Manufacturing  TRL,  Biomedical  TRL 
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Technology  Readiness  Levels 
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System  Test,  Launch 
&  Operations 


r\ 

TRL  9 


9  -  Actual  system  “flight  proven”  through  successful 
mission  operations 


System/Subsystem 

Development 


TRL  8 

TRL  7 


Technology 

Demonstration 


Technology 

Development 


Research  to  Prove 
Feasibility 


TRL  6 

TRL  5 


TRL  4 

TRL  3 


8  -  Actual  system  completed  and  “flight  qualified” 
through  test  and  demonstration 

7  -  System  prototype  demonstration  in  a  operational 
environment 

6  -  System/subsystem  model  or  prototype 
demonstration  in  a  relevant  environment 

5  -  Component  and/or  breadboard  validation  in 
relevant  environment 

4  -  Component  and/or  breadboard  validation  in 
laboratory  environment 

3  -  Analytical  and  experimental  critical  function  and/or 
characteristic  proof-of-concept 


Basic  Technology 
Research 


2 

1 


Technology  concept  and/or  application  formulated 


Basic  principles  observed  and  reported 
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Software  Readiness  Levels  (SWRL) 
Missile  Defense  Agency  (MDA) 
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Engineering  Manufacturing  Readiness  Levels  (Hardware) 

•  EMRL 1  •  SWRL  1 

-  Breadboard  -  Concept 


EMRL  2 

-  Prototype 

EMRL  3 

-  Advanced  development 
EMRL  4 

-  Similar  production 


SWRL  2 

-  Prototype 


SWRL  3 

-  Development 

SWRL  4 

-  Functional 


EMRL  5  *  SWRL  5 

_  frp  -  Deployable 

Software  Readiness  Levels 


J 


Capability 

Demonstration 


1 


SWRL  5 


DR2 


I  SWRL  4 


DR1 


SWRL  2 


Concept 

Design 
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DUSD(  S&T)  TRA  Deskbook  2005 
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Technology  Readiness  Level 

Software  Description 

1.  Basic  principles  observed  and  reported 

Lowest  level  of  software  technology  readiness.  A  new  software  domain  is  being  investigated  by  the  basic 
research  community.  This  level  extends  to  the  development  of  basic  use,  basic  properties  of  software 
architecture,  mathematical  formulations,  and  general  algorithms. 

2.  Technology  concept  and/or  application 
formulated 

Once  basic  principles  are  observed,  practical  applications  can  be  invented.  Applications  are  speculative,  and 
there  may  be  no  proof  or  detailed  analysis  to  support  the  assumptions.  Examples  are  limited  to  analytic  studies 
using  synthetic  data. 

3.  Analytical  and  experimental  critical 
function  and/or  characteristic  proof  of 
concept 

Active  R&D  is  initiated.  The  level  at  which  scientific  feasibility  is  demonstrated  through  analytical  and  laboratory 
studies.  This  level  extends  to  the  development  of  limited  functionality  environments  to  validate  critical 
properties  and  analytical  predictions  using  nonintegrated  software  components  and  partially  representative 
data. 

4.  Module  and/or  subsystem  validation 
in  a  laboratory  environment  (i.e.,  software 
Prototype  development  environment). 

Basic  software  components  are  integrated  to  establish  that  they  will  work  together.  They  are  relatively  primitive 
with  regard  to  efficiency  and  robustness  compared  with  the  eventual  system.  Architecture  development 
initiated  to  include  interoperability,  reliability,  maintainability,  extensibility,  scalability,  and  security  issues. 
Emulation  with  current/legacy  elements  as  appropriate.  Prototypes  developed  to  demonstrate  different  aspects 
of  eventual  system. 

5.  Module  and/or  subsystem  validation  in  a 
relevant  Environment 

Level  at  which  software  technology  is  ready  to  start  integration  with  existing  systems.  The  prototype 
implementations  conform  to  target  environment  /  interfaces.  Experiments  with  realistic  problems.  Simulated 
interfaces  to  existing  systems.  System  software  architecture  established.  Algorithms  run  on  a  processor(s)  with 
characteristics  expected  in  the  operational  environment. 

6.  Module  and/or  subsystem  validation 
in  a  relevant  end-to-end  environment) 

Level  at  which  the  engineering  feasibility  of  a  software  technology  is  demonstrated.  This  level  extends  to 
laboratory  prototype  implementations  on  full-scale  realistic  problems  in  which  the  software  technology  is 
partially  integrated  with  existing  hardware/software  systems 

7.  System  prototype  demonstration 

in  an  operational  high-fidelity  environment 

Level  at  which  the  program  feasibility  of  a  software  technology  is  demonstrated.  This  level  extends  to 
operational  environment  prototype  implementations  where  critical  technical  risk  functionality  is  available  for 
demonstration  and  a  test  in  which  the  software  technology  is  well  integrated  with  operational 
hardware/software  systems. 

8.  Actual  system  completed  and  mission 
qualified  through  test  and  demonstration  in 
an  operational  environment 

Level  at  which  a  software  technology  is  fully  integrated  with  operational  hardware  and  software  systems. 

Software  development  documentation  is  complete.  All  functionality  tested  in  simulated  and  operational 
scenarios.  _  -  __ 

9.  Actual  system  proven  through  successful 
mission-proven  operational  capabilities 

Level  at  which  a  software  technology  ris  readilv  repeatable  and  reusable?  The  software  based  on  the 
technology  is  fully  integrated  with  operatioh^t“haFdwaFe/softwafe"Systems.  All  software  documentation  verified. 
Successful  operational  experience.  Sustaining  software  engineering  support  in  place.  Actual  system. 
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Software  TRL  Limitations 
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•  Software  differs  from  hardware  in  that  taking  an 
operational  product  and  using  it  in  a  new  context  or 
system  does  not  necessarily  correlate  to  system  success  in 
performance  or  in  achieving  planned  cost  and  schedule 
benefits 

-  I  n  some  situations  it  may  introduce  more  complications  and 
problems  than  if  the  code  was  not  reused 


•  TRLs  inherently  assume  "good 
reuse" 

-  I  ncreased  dependability 

-  Reduce  product  and  process  risk 

-  Accelerated  development 


TRLs  do  not  adequately  address  "bad 
reuse"  or  COTS/GOTS  and  OSS 

-  Obsolescence 

-  Breakage 

-  Requirements  and  usage  differences 

-  Unfamiliarity 
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Software  Reuse  Root  Cause  Analysis 
Six  Sigma  Project  #1299 


Reusable  components  Product  requirements 

Domain  not  applicable  v  Changed  -  volatile 


Design  not  compatible 

Code  not  compatible 
Processors  not  compatible 
Quality  insufficient 


Poor  quality 


More  stringent  testing 


Quality  inspection 


Turnaround  time 


Automation  and  tools 


Distributed  development 


Staff  familiarity  with  reuse  SW. 


Reuse  decision  makers. 


Process  maturity 


Development 

tools/facilities 


Development 

staff/process 


Time  of  reuse  decisions 


Level  of  cost  adjustment 


Estimation 


SW  reuse  is  less 
than  planned 
because  of: 
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RRL  Background 

Software  Engineering  Institute  (SEI) 
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•  TRL  for  Non- Developmental  Item  Software  (Smith  2004) 

-  Requirements  Satisfaction 

•  Rates  how  well  requirements,  including  functional  (e.g.,  throughput,  accuracy, 
latency  )  and  non-functional  (e.g.,  reliability,  maintainability)  are  allocated  to  a 
given  software  product  or  technology  to  be  satisfied  by  it. 

•  Accounts  for  the  number  of  requirements  are  satisfied  as  well  as  any  provided 
functionality  that  is  not  required 

-  Environmental  Fidelity 

•  Addresses  how  faithfully  the  development  environment  of  the  software  asset  has 
been  demonstrated  to  operate  in  the  target  operational  environment. 

-  Product  Criticality 

•  The  degree  to  which  the  target  system  is  dependent  upon,  or  inseparable  from 
the  product  or  technology. 

-  Product  Aging 

•  The  availability  of  the  product  over  its  lifespan  relative  to  the  requirements  of 
the  system  under  development 

-  Product  Maturity 

•  Maturity  of  the  software  product  or  technology  relative  to  three  distinct 
modes/domains:  COTS,  GOTS,  OSS 
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RRL  Background 
SEI  (page  2) 
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•  I  mpACT  Methodology  for  COTS  (Smith  2005) 

-  I  mportance 

•  Criticality  to  the  system;  difficulty  of  effecting  a  work-around  if  the  technology  or 
product  doesn't  work  (or  isn't  available) 

-  Availability 

•  The  degree  to  which  the  product  or  technology  is  commercially  available 

-  Capability 

•  The  functional  fit  (or  misfit)  between  the  product  or  technology  and  the 
requirements  of  the  system 

-  Timeframe 

•  A  measure  of  how  the  lifecycle  of  the  product  or  technology  matches  the 
lifecycle  for  the  system.  Will  it  be  available  when  needed?  Over  the  life  of  the 
system? 
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RRL  Background  „onT„no, 

National  Aeronautics  and  Space  Agency  (NASA) 


•  NASA  Earth  Science  Data  Systems  (ESDS)  Software  Reuse  Working 
Group  (Wolfe,  Marshall,  2007-2008) 

-  Determine  reuse  maturity  of  software  assets  being  prepared  for  reuse 

-  I  nitially  developed  for  the  Earth  science  domain,  applicable  to  general 

-  Promote,  facilitate,  catalog  and  incentivize  reuse 

-  Reuse  Enablement  System 

•  Web- based  portal.  Reuse  metadata  of  an  existing  software  asset 

•  Aligned  with  familiar  1-9  scale  TRL 

•  Topic  Areas 

-  Portability 

-  Extensibility 

-  Documentation 

-  Support 

-  Packaging 

-  Intellectual  Property 

-  Standards  Compliance 

-  Verification  and  Testing 

-  Modularity 


13 


NORTHROP  GRUMMAN  CORPORATION  © 


14 


NASA  RRL  Topic  Areas  and  Rating  Scale 
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Portability 

Extensibility 

Documentation 

Support 

Packaging 

Intellectual  Property 
issues 

Standards  compliance 

Verification  &  Testing 

Modularity 

Level 

1 

The  software  is 
not  portable  at 
any  cost 

No  ability  to  extend  or 
modify  program  behavior 

Limited  internal 
documentation 
available 

No  support 
available 

Source  code 
available 

Potential  owners  and 
stakeholders  of  product 
have  been  identified. 

Follows  no  particular 
standard 

No  testing  performed 

No  designs  for 
modularity  or  reuse 

Level 

2 

Some  parts  of 
the  software  may 
be  portable 

Prohibitive  costs  and 
efforts  need  to  modify  or 
extend  the  system 

Fully  commented 
source  code 
available 

Known  contact 
available 

Relevant  intellectual 
policies  of  potential  owners 
and  stakeholders  have 
been  reviewed. 

Follows  some  parts  of 
common  standards  and 
best  practices 

Software  application 
formulated  and  unit 
testing  performed 

Level 

3 

The  software  is 
only  portable  with 
significant  costs 

Can  be  extended  with 
the  input  of  considerable 
time  and  effort  on  par 
with  recreating  system 
separately 

Basic  external 
documentation 
available 

Original 
developers 
provide  proactive 
support 

Detailed 

installation 

instructions 

available 

Intellectual  property 
agreements  have  been 
proposed  to  potential 
stakeholders. 

Follows  a  company-wide 
standard  for  development 
and  testing 

Testing  includes  testing 
for  error  conditions  and 
proof  of  handling  input 
errors 

Modularity  at  major 
system  or  subsystem 
level  only 

Level 

4 

The  software 
may  be  portable 
at  a  reasonable 
cost 

Can  be  modified  and 
extended  through 
configuration  changes, 
minimal  modification  of 

source 

Reference  manual 
available 

Latest  updates  or 
patches  are 
available  but  not 
very  frequently 

Potential  stakeholders  have 
negotiated  on  intellectual 
property  agreements  and 
authorship  issues. 

Most  components  follow 
a  complete,  universal 
standard,  but  not 
validated 

Software  application 
demonstrated  in  a 
laboratory  environment 

Level 

5 

The  software  is 

moderately 

portable 

Consideration  for  future 
extensibility  designed 
into  system,  extensibility 
approach  somewhat 
defined 

User  manual 
available 

Informal  user 

community 

available 

Software  is  easily 
configurable  for 
different 
environments 

Agreement  and  approval  on 
authorship,  attribution,  and 
intellectual  property  issues 
has  been  obtained  from 
stakeholders. 

All  components  follow  a 
universal  standard,  but 
only  partially  validated 

Software  application 
tested  and  validated  in  a 
laboratory  environment 

Partial  segregation  of 
generic  and  specific 
functionality 

Level 

6 

The  software  is 
portable 

Designed  from  the  start 
to  allow  easy 
extensibility,  provides 
many  points  of 
extensibility  and  a 
thorough  and  detailed 
extensibility  plan 

Tutorials  available 

Centralized 
support  available 

Authorship,  attribution,  and 
intellectual  property 
statements  have  been 
drafted  to  reflect  agreement 
among  stakeholders  on 
intellectual  property  and 
authorship. 

Validated  to  follow  a 
specific  proprietary 
standard 

Software  application 
demonstrated  in  a 
relevant  environment 
(Earth  science  related) 

Level 

7 

The  software  is 
highly  portable 

Proven  to  be  extensible 
internally,  code 
structured  to  provide 
loose  coupling  and  high 
cohesion 

Interface  guide 
available 

Organized/define 
d  support  by  the 
original  developer 
available 

OS  detect  and 
auto-build  for 
supported 
platforms 

Authorship  and  intellectual 
property  statements 
included  in  product 
prototype. 

Validated  to  comply  to  a 
specific  open  standard 

Software  application 
tested  and  validated  in  a 
relevant  environment 
(Earth  science  related) 

Clear  delineations  of 
specific  and  reusable 
components 

Level 

8 

Proven  extensibility  on  a 
major  external  program, 
provides  a  clear  plan  for 
modifying  and  extending 
features 

Extension  guide 
and/or 

Design/Developme 
nt  guide  available 

Support  by 

organization 

available 

Manifestation  of  authorship, 
attribution,  and  intellectual 
property  statements 
reviewed  in  product 
prototype  before  product 
release. 

Proven  by  validation  to 
comply  with  a  “gold” 
standard 

Software  application 
"qualified"  through  test 
and  demonstration 
(meets  requirements) 
and  successfully 
delivered  to  the  Earth 
science  environment 

Level 

9 

The  software  is 

completely 

portable 

Proven  extensibility  in 
multiple  scenarios, 
provides  specific 
documentation  and 
features  to  build 
extensions 

Full  software 
lifecycle 

engineering  design 

documentation 

available 

Large  user 
community  with 
well-defined 
support  available 

GUI  installation 

environment 

provided 

Reviewed  authorship, 
attribution,  and  intellectual 
property  statements 
packaged  with  product  for 
release. 

“Gold”  standard 
compliance  of  entire 
system  and 
development, 
independently  validated 

Actual  software 
application  tested  and 
validated  through 
successful  use  of 
application  output 

All  functions  and  data 
encapsulated  into 
objects  or  accessible 
through  web  service 
interfaces 
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Northrop  Grumman  ( NGC)  Reuse 
Readiness  Level  Framework 
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•  NGC  is  developing  Reuse  Readiness  Levels  ( RRL)  as  a  decision 
framework  to  evaluate  the  technical  viability  of  leveraging  existing 
software 

-  Merges  the  TRL  concept  with  NGC's  Decision  Analysis  and  Resolution  (DAR) 
process 

•  Aligned  with  the  1-9  ascending  TRL  scale 

•  DAR 

-  Reduces  subjectivity,  increases  rigor  and  consistency 

-  Encourages  disciplined  objective  thinking  and  stakeholder  buy-in  via  evidence 

-  Ensures  best  possible  solutions  for  high  risk  decisions 

-  Avoids  premature  commitment  to  a  point  design 

-  Flexible  -  fits  all  situations 

•  Multi-attribute  /  multivariate  considered 

•  DAR  allows  tailoring 

•  Applicable  to  product  line,  non-product  line,  COTS,  GOTS,  NDI,  OSS,  etc. 
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NGC  Reuse  Readiness  Attributes 
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•  Resources  •  Compatibility 

-  Supporting  processes  and  resources  _  platform  compatibility 

-  Software  familiarity  _  version  compatibility 

-  Developer  experience  _  Language  compatibility 


•  Readability 

-  Quantity  and  level  of  documentation 

-  Accuracy  and  completeness  of 
documentation 

•  Usability 


Tailoring  /  Rework 

-  Restructuring  /  Re-factoring 

-  Re-engineering 

-  Re- implementation 

-  Re- integration  and  Re- test 


Configurability,  Openness  and 
Modularity 

Extensibility 

Scalability 

Well-defined  and  stable  interfaces 


Transportability 

-  Architecture  /  design 
synchronization 

-  Percentage  of  translation  to 
new  context 


•  Maturity 

-  Product  life  cycle  stage 

-  Maintenance 


I  ndex  of  new  requirements 
incorporation 
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# 

Category 

Attribute 

Description 

1 

Resources 

Supporting  Processes  and 
Resources 

The  consonance  of  the  development  methods  and  activities  to  the 
integration  of  the  reuse  software  in  the  new  context/system  as  well  as 
the  accessibility  and  availability  of  expertise  related  to  the  reuse 
software  (either  internal  or  external  to  the  organization). 

2 

Resources 

Software  Familiarity 

The  level  of  understanding  and  practice  that  the  development  team  has 
in  working  with  the  reuse  software. 

3 

Resources 

Developer  Experience 

The  knowledge,  skill,  proficiency  and  expertise  of  the  development 
team  within  the  system  domain. 

4 

Readability 

Quantity  and  Level  of 
Documentation 

The  amount  and  the  detail  of  available  descriptions  of  the  software  such 
as:  annotation  in  the  code,  reference  manuals,  style  guides,  developer 
user  guides,  use  cases,  etc. 

5 

Readability 

Accuracy  and  Completeness 
of  Documentation 

The  degree  to  which  the  reuse  software  documentation  is 
comprehensive,  usable  and  reliably  describes  and  explains  the  product. 

6 

Usability 

Configurability,  Openness 
and  Modularity 

The  extent  to  which  the  reuse  software  may  be  added,  upgraded  and 
have  its  components  replaced;  as  well  as  the  efficient  separation  of 
system  concerns  realized  through  the  logical  boundaries  between 
components. 

7 

Usability 

Extensibility 

The  ability  of  the  system  to  accommodate  future  growth  either  through 
the  addition  of  new  functionality  or  through  the  modification  of  existing 
functionality  while  minimizing  the  impact  to  other  existing  system 
functions  or  infrastructure. 

8 

Usability 

Scalability 

The  degree  to  which  the  design  of  the  reuse  software  handles 
increasing  amounts  of  work,  data,  throughput,  quantities,  resources, 
etc.  with  graceful  or  no  degradation  in  performance. 
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# 

Category 

Attribute 

Description 

9 

Usability 

Well-defined  and  Stable 
Interfaces 

The  clarity,  understandability  and  integrity  of  the  reuse  software  (internal 
and  external)  interfaces  as  well  as  the  robustness  of  the  interfaces  under 
changing,  stressing  or  anomalous  conditions 

10 

Maturity 

Product  Life  Cycle 

Stage 

The  current  point  in  the  reuse  software's  evolution  (ranging  from 
"bleeding  edge"  new  to  obsolete)  and  the  degree  to  which  it  has  been 
tried,  tested  and  proven  in  a  working  system.  Factors  to  consider: 
usage  and  acceptance  in  the  domain  and  the  Industry 

11 

Maturity 

Maintenance 

The  required  resources  to  upkeep  of  the  reuse  software  for  correcting 
faults  and  keeping  it  operational.  Factors  to  consider:  Software  Problem 
Report  history,  number  and  frequency  of  software  patches,  etc. 

12 

Compatibility 

Platform  Compatibility 

The  degree  to  which  the  original  hardware  architecture  and  software 
framework  on  which  reuse  software  runs  is  similar  or  complimentary  to 
the  new  context/system.  Factors  to  consider:  computer  architecture, 
operating  system,  graphical  user  interface,  etc. 

13 

Compatibility 

Version  Compatibility 

The  level  at  which  the  reuse  software  behaves  in  the  intended  and 
expected  manner  when  it  interacts  with  the  other  software  components, 
products,  tools,  environments  and  platforms  in  the  new  context/system. 
Factors  to  consider:  rate  of  change/upgrades  of  underlying  products, 
frequency  of  synchronization  points,  etc. 

14 

Compatibility 

Language  Compatibility 

The  extent  to  which  the  programming  set  of  instructions  of  the  reuse 
software  requires  translation,  reimplementation,  or  re-compilation  in 
order  to  work  in  the  new  context/system 
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# 

Category 

Attribute 

Description 

15 

Tailoring  / 
Rework 

Restructuring  /  Re¬ 
factoring 

The  extent  to  which  the  existing  software  needs  to  be  cleaned  up  -  i.e.; 
improve  its  understandability;  remove  extraneous  (dead)  code,  make 
the  internal  structure  and  design  more  efficient,  maintainable  and 
amenable  to  change,  etc. 

16 

Tailoring  / 
Rework 

Re-engineering 

The  amount  of  reverse  engineering  or  learning  required  to  modify  the 
design  for  integration  in  the  new  context/system. 

17 

Tailoring  / 
Rework 

Re-implementation 

The  amount  of  adaptation  of  the  existing  code  and/or  the  addition  of 
new  code  to  meet  the  objectives  and  environment  of  the  new 
context/system 

18 

Tailoring  / 
Rework 

Re-integration  and 
Re-test 

The  effort  to  combine  the  existing  software  into  the  new  context/system 
and  verify  that  resulting  product  functions  within  performance,  reliability, 
and  other  criteria  in  the  new  system/context 

19 

Transportability 

Architecture/Design 

Synchronization 

The  degree  of  similarity  of  the  structure  in  which  the  reusable  software 
will  interact  in  the  new  context/system.  Factors  to  consider:  reuse  of 
an  entire  product  or  functional  components;  control  mechanisms,  data 
exchange,  logical  dependencies 

20 

Transportability 

Percentage  of 
Translation  to  New 
Context 

The  percentage  change  in  the  behavior,  conditions  and/or  constraints 
in  which  the  reuse  software  will  operate  in  the  new  context/system. 
Factors  to  consider:  operational  scenarios,  operational  threads,  use 
cases,  etc. 

21 

Transportability 

Index  of  New 

Requirements 

Incorporation 

The  ratio  of  component  level  requirements  allocated  to  the  reuse 
software  that  are  new  relative  to  a  normalized  measure  of  the 
requirements  that  are  already  fully  and  partially  satisfied  by  the 
software 
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Comparison  of  Reuse  Attributes 


NORTHROR  GRUMMAN 


NGC  RRL 

NASA  RRL 

Army  SW  TRL 

SEI  NDI 

Resources  | 

Supporting  Processes  and  Resources 

Support 

Development  Process 

Software  Familiarity 

Developer  Experience 

Readability  | 

Accuracy  and  Completeness  of  Documentation 

Documentation 

Previous  System  Documents  /  Code 

Level  of  Documentation 

Usability  | 

Open  Architecture  /  Modularity 

Modularity 

Configurability  and  Openness 

Portability 

Extensibility  across  Platforms 

Extensibility 

Scalability 

Well-defined  and  Stable  Interfaces 

Standards  Compliance 

[Compatibility  | 

Platform  Compatibility 

Packaging 

Development  Environment 

Environment  Fidelity 

Version  Compatibility 

Test  (Verify)  Environment 

Language  Compatibility 

|  Maturity 

Years  in  Operation 

Verification  and  Testing 

Technology  Prototyped/  Used  Existing  System 

Maintenance 

Open  Problem  Reports 

Maturity 

Upgrades  /  Technology  Insertion 

|  Rework  | 

Restru  ct  u  ri  n  g/Refacto  ring 

Change  To  Code 

Re-engineering 

Re-implementation 

Re-integration  and  Re-test 

[Transportability  | 

Number  of  Contexts/Instantiations  in  which  reused 

Studies  /  Test  Use  Results 

Architecture/Design  Synchronization 

Technology  Critical 

Percentage  of  Translation  to  New  Context 

|  Other 

Index  of  New  Requirements  Incorporation 

Precision  /  Performance 

Requirements  (Functional 
and  Non-Functional) 

Intellectual  Property 

Availability 

Safety  /  Security 
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Standard  Process  Manual  944  - 
Decision  Analysis  and  Resolution 


NORTHROR  GRUMMAN 


•  Sample  Methodologies 

-  Trade  Study 

-  Simple  Multi- Attribute  Rating  Technique  (SMART) 

-  Analytical  Hierarchy  Process  (AHP) 

-  Pay-off  table  with  application  of  an  analysis  technique  (MiniMax,  Expected 
Value,  MaxiMax,  Minimum  Regret,  etc.) 

-  Decision  Trees  and  I  nfluence  Diagrams 

-  Simulation 

-  Group  Techniques  (e.g.,  Delphi) 
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Reuse  Readiness  Levels  -  NGC  Approach 


NORTHROP  GRUMMAN 


Identify 

Candidates 


Date  Contacted: 


Reference  Information 

Company  Name 

Company  Address 

Contact  Name 

Contact  Phone  Number 

Vendor  Sales 
Representative 

Obtain 
Pertinent 
Data  / 

Information 


Product  Information 


Advice  or  Warnings 


Unexpected  Benefits 
Opinions  on  Other 
Products 
Version  Used 


Tailor 

Candidate 

Attributes 


Assign 
Weights 
Specific  to 
Situation 


Sensitivity  analysis:  Green  =  1st  highest  score,  Yellow  =  2nd  highest  score,  Red  =  3rd  highest  score 


No. 

Evaluation  Criteria 

Evaluation 

Method 

Scoring 

Function 

Reuse  Asset  1 

COTS  A 

Weiqht  (%) 

Score 

Weighted 

Score 

Score 

Weighted 

Score 

1 

Supporting  Processes  and  Resources 

2.0% 

2.0 

0.04 

4.2 

0.08 

2 

Software  Familiarity 

2.0% 

9.8 

0.20 

4.8 

0.10 

3 

Developer  Experience 

10.0% 

1.6 

0.16 

3.5 

0.35 

4 

Quantity  and  Level  of  Documentation 

1 .0% 

0.1 

0.00 

8.2 

0.08 

5 

Accuracy  and  Completeness  of  Documentation 

1 .0% 

1.3 

0.01 

3.0 

0.03 

6 

Configurability,  Openness  and  Modularity 

17.0% 

4.0 

0.67 

'  5.2 

0.89 

7 

Extensibility 

8.0% 

7.8 

0.62 

5.5 

0.44 

8 

Scalability 

5.0% 

8.1 

0.41 

6.5 

0.33 

9 

Well-defined  and  Stable  Interfaces 

12.0% 

1.8 

0.22 

1.0 

0.12 

10 

Product  Life  Cycle  Stage 

5.0% 

2.3 

0.12 

1.7 

0.09 

11 

Maintenance 

3.0% 

7.6 

0.23 

9.0 

0.27 

12 

Platform  Compatibility 

0.0% 

0.0 

0.00 

0.0 

0.00 

13 

Version  Compatibility 

7.0% 

5.1 

0.36 

2.1 

0.15 

14 

Language  Compatibility 

0.0% 

4.3 

0.00 

2.0 

0.00 

15 

Restructuring  /  Re-factoring 

6.0% 

2.2 

0.13 

3.0 

0.18 

16 

Re-engineering 

0.0% 

0.0 

0.00 

0.0 

0.00 

17 

Re-implementation 

4.0% 

9.1 

0.36 

3.0 

0.12 

18 

Re-integration  and  Re-test 

4.0% 

2.0 

0.08 

2.0 

0.08 

19 

Architecture/Design  Synchronization 

8.0% 

1.8 

0.15 

1.5 

0.12 

20 

Percentage  of  Translation  to  New  Context 

0.0% 

4.0 

0.00 

3.5 

0.00 

21 

Index  of  New  Reguirements  Incorporation 

5.0% 

3.8 

0.19 

2.7 

0.14 

TOTAL: 

100.0% 

3.94 

|  3.55 

A 


Translate 
into  RRL 


RRL 

Description 

1 

Not  reusable  in  the  given  context 

2 

Not  practical  to  reuse  in  the  given  context 

3 

Conceptual  reuse  possible;  significant  risk  to 
implementation 

4 

Reuse  with  6  or  more  attributes  of  concern  - 
assume  substantial  cost  and  risk 

5 

Reuse  with  3  to  6  attributes  of  concern  - 
assume  a  reasonable  amount  of  cost  and  risk 

6 

Reuse  with  1  to  3  attributes  of  concern  - 
potential  for  some  cost  and  risk 

7 

High  reuse  with  minimum  cost  and  risk 

8 

Demonstrated  reuse  by  multiple  adopters 

9 

Proven  serial  reuse  by  multiple  skills  and 
experience  level 
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Outcomes 


NORTHROR  GRUMMAN 


•  Decisions  /  Assessments 

-  Technical  viability  and  rank  order  of  reuse  candidate  software 

-  J  ustification  not  to  reuse 

-  I  n vestment  in  maturing  a  potential  reuse  asset 

-  Use  as  a  component  to  determine  an  overall  for  Software  TRL  of  a  critical 
technology 

•  I  nsight 

-  Understanding  of  the  level  risk  associated  with  incorporating  software 
technologies  into  a  system  or  solution 

-  Sensitivity  of  driving  factors  that  affect  reuse  success 

-  Degree  of  modification  (effort)  required  to  reuse  the  product 

•  I  mproved  size  and  cost  estimates 
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